Adding support for sharing memory between the module and the engine

[yairgott]

Overview

Sharing memory between the module and engine reduces memory overhead by eliminating redundant copies of stored records in the module. This is particularly beneficial for search workloads that require indexing large volumes of documents.

Vectors

Vector similarity search requires storing large volumes of high-cardinality vectors. For example, a single vector with 512 dimensions consumes 2048 bytes, and typical workloads often involve millions of vectors. Due to the lack of a memory-sharing mechanism between the module and the engine, ValkeySearch currently doubles memory consumption when indexing vectors, significantly increasing operational costs. This limitation introduces adoption friction and reduces ValkeySearch's competitiveness.

Technical Details

Memory Allocation Strategy

At a fundamental level, there are two primary allocation strategies:

• [Chosen] Module-allocated memory shared with the engine.
• Engine-allocated memory shared with the module.

For ValkeySearch, it is crucial that vectors reside in cache-aligned memory to maximize SIMD optimizations. Allowing the module to allocate memory provides greater flexibility for different use cases, though it introduces slightly higher implementation complexity.

Shared SDS

Shared SDS, a new data type, facilitates module-engine memory sharing with thread-safe intrusive reference counting. It preserves SDS semantics and structure while adding ref-counting and a free callback for statistics tracking.

New Module APIs

• VM_CreateSharedSDS:
  • Creates a new Shared SDS.
  • Accepts an allocation function for fine-grained control (e.g., cache alignment).
  • Accepts a free callback function to track deallocations.
• VM_SharedSDSPtrLen: Retrieves the raw buffer pointer and length of a Shared SDS.
• VM_ReleaseSharedSDS: Decreases the Shared SDS ref-count by 1.

Extended Module APIs

• VM_HashSet: Supports setting a shared SDS in the hash.
• VM_HashGet: Retrieves a shared SDS and increments its ref-count by 1.

Engine Hash Data-Type

ValkeySearch indexes documents which reside in engine as t_hash data-type records. While JSON is also supported, it is out of scope for this discussion. The t_hash implementation is based on either list-pack for small datasets or hashtable for larger ones.

Since list-pack performs deep copies, it cannot support intrusive ref-counting semantics. As a result, if list-pack is used as the underline data-type while setting a shared SDS, .e.g. by calling VM_HashSet, it is converted to hashtable. Additionally, for the same reason, a shared SDS is never stored as embedded value in hashtable entry.

[yairgott]

Planning to add unit tests soon. Publishing early to start the discussion rolling.

[codecov]

Codecov Report

Attention: Patch coverage is 20.21277% with 75 lines in your changes missing coverage. Please review.

Project coverage is 70.94%. Comparing base (3f6581b) to head (f9aad1a).

Files with missing lines	Patch %	Lines
src/module.c	6.97%	40 Missing ⚠️
src/sds.c	23.33%	23 Missing ⚠️
src/sds.h	0.00%	8 Missing ⚠️
src/t_hash.c	69.23%	4 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #1804      +/-   ##
============================================
- Coverage     70.99%   70.94%   -0.05%     
============================================
  Files           123      123              
  Lines         65651    65727      +76     
============================================
+ Hits          46609    46631      +22     
- Misses        19042    19096      +54     

Files with missing lines	Coverage Δ
src/server.h	100.00% <ø> (ø)
src/t_hash.c	95.71% <69.23%> (-0.52%)	⬇️
src/sds.h	75.29% <0.00%> (-7.61%)	⬇️
src/sds.c	83.60% <23.33%> (-3.08%)	⬇️
src/module.c	9.61% <6.97%> (+<0.01%)	⬆️

... and 14 files with indirect coverage changes