Do old snapshots ever get purged?

[natilivni]

I want to build a completely in memory cluster but I am not clear about the snapshot mechanisms. I understand that they are there to compact the log, however, they must still take up space. Won't a long running service eventually run out of memory? Do these snapshots ever get purged from the audit log?

[sakno]

There are two audit logs supported out-of-the-box:

1. ConsensusOnlyState is suitable if you need only the consensus across the cluster. It allocates on the heap. However, it doesn't maintain the content of the log entries. Committed log entries are always squashed to the simple representation containing the index of the last committed log entry and its term. So this type of log never grows in memory.
2. PersistentState doesn't store the log entries in memory. Instead, it uses persistent storage such as disk.

Persistent log is highly optimized for disk I/O, parallel reads, in-memory cache for fast lookup and many other things. Of course, you need to configure it according with your needs. The default implementation cannot understand the semantics of the custom log entries. Without this knowledge it cannot construct snapshots automatically. However, it provides extensibility point which gives you opportunity to implement interpreter of the log entries and explain semantics of them to the persistent log. In this case, snapshots become possible. Let's assume that your log contains the following records:

(1) x := 10
(2) y := 20
(3) x := 30

We can compress such log as follows:

(2) y := 20
(3) x := 30

I recommend you to inspect example of Raft-based cluster which is available in my repo. The sample project contains everything you need:

1. Demonstration of various transports: HTTP, TCP, UDP
2. Consensus-only cluster
3. Replication cluster with persistent state
4. Trivial interpreter of the log entries which is required for snapshotting

Here is a cookbook for persistent storage.

[natilivni]

If the ConsensusOnlyState does not maintain the content of the log entries, how does a new node which comes online replay the audit log in order to build its correct current state? Would you have to use persistent state for such functionality to work?

[sakno]

It can't. ConsensusOnlyState according with it's name is for consensus only, i.e. leader election. It cannot be used for replication. Consensus-only scenario requires to maintain term, last index, last commited index. In other words, just metadata of the log entries, not their content. The volume of metadata is predictable and has fixed size in memory.