Looking for health check suggestions.

[natilivni]

I need to establish a health check endpoint for each node in my cluster. I am looking for suggestions regarding things to check. Any suggestions would be appreciated. All I can think of thus far is checking connectivity to other nodes.

[sakno]

Checking connectivity between the nodes is not a primary goal for health check, IMO. It's responsibility of the monitoring tool configured to check the health of the underlying network infrastructure.

If you need to perform application-level checks then this article will be useful.

[natilivni]

I guess my question is: outside of things specific to my implementation, when is a cluster node considered degraded or unhealthy?

[sakno]

Broadcast time spent by the leader node to send the heartbeats to the followers should not be greater than lower election time. You can use DotNext.Net.Cluster.Consensus.Raft.MetricsCollector to check this time. Frequent re-elections are evidences of abnormal behavior of the cluster.

Other things such as memory consumption, GC cycles, CPU utilization are common and not specific to the current Raft implementation.