[Navy Cache] how are non-graceful shutdowns handled?

[dannycho7]

Specifically for the block cache, I am curious to learn how non-graceful shutdowns are handled.

From reading the code, it appears that the metadata for BlockCache (Index and RegionManager state) is persisted upon shutdown. I cannot seem to find any other triggers for the persistence of metadata state. On the other hand, regions can be flushed once there is no longer enough free space to service an allocation. It seems possible that the persistence of the metadata state does not complete successfully, causing there to be skew between the metadata and regions (e.g. server crashes/killed).

There are two scenarios I would expect to happen when the metadata state is not in sync with the flushed regions:

1. Data "loss" since entries are no longer searchable through the index (or rather, the index may just point to invalid addresses).
2. Since Index entries appear to only be removed when a region is being reclaimed, it seems like it's possible for some index entries to not be removable until the same key is written again (the ones that belonged to regions that were flushed but the updated index was not persisted).

Is my understanding correct or am I misunderstanding something? If my conclusions are correct I had a few questions:

1. What is the rationale for not persisting the metadata periodically to reduce the amount of skew?
2. Are there any considerations for getting rid of those non-removable index entries due to this issue?
3. Related to (2): Is there any way to instruct NavyCache to limit its memory usage for metadata? If not, are there any considerations one should keep in mind when configuring the HybridCache to avoid OOMs?

[haowu14]

CacheLib serves as a cache, not a storage. So CacheLib does not support transactions and it has no guarantees for non-graceful shut down. If that happens we advise our users to clear the cache.

limit its memory usage for metadata

For bighash, the overhead is pretty much the bloom filters which can be bounded.
For blockcache, you can estimate the RAM overhead of each item is about 16 bytes.

Let's say you have 1TB flash space and decided to use ~400GB bighash and 600GB block cache and you've set the threshold for large item to be 1KB.

In bighash, each bucket has a bloom filter, and say you have 4 hashes and 8 bits for each hash table that's 4 bytes per bucket. Each bucket is 4KB, which means the DRAM overhead is 1/1024 of the flash space. For 400GB bighash, that means 400MB RAM.

In blockcache, in the worst case all the items are 1KB and each item has 16 bytes overhead, that's 1/64. For 600GB, that' around 9GB.
There are other overhead from the region, but those are pretty small compared to this.

In production, it's better to look at the actual number of items that end up in blockcache and reestimate the overhead since it's unlikely all items being 1KB.

[dannycho7]

Thank you, this is super helpful!

[dannycho7]

Follow-up: I understand that CacheLib is a cache and does not guarantee persistence, but it still seems like a desirable property of a cache to limit the frequency and size of data that is dropped (to avoid the cost of rewarming the cache). Are non-graceful shutdowns rare in practice for existing CacheLib users where this is not a concern?