Integrated Storage - leader node is acting up

[daipok]

For some reason on my cluster of 3 nodes recently the node1 (the leader) is acting up. I find the the write speed is degrading (read not as bad, but degrading as well). I tried to step it down it does not step down (and I forgot to capture the log).

I stopped it on node 1 so essentially the leader passes to node 2.

I was trying to see, as node 1 is the follower now, is there anyway that I can "reinstate" it by doing:

1. Remove the raft data path from filesystem
2. Force a resync to happen from leader node?

I tried to remove it from node2 doing a remove-peer, that works, but no longer I can rejoin node1 again to the cluster because I removed the raft data path on filesystem on node1.

In this kind of situation, anything I can do to save this node1 back?

[daipok]

I got the sync working and the follower has synced the data.

However, for some reason one of the nodes fail to step down.

I found that with my set up of node1,2,3, when I step down from node1 the leadership can go to node2. From node2 when I do step down, it seems that node3 should be next, but as there's some weird tls error between node2 and node3, the leadership shift did not happen, and thus node2 once again grabs the leadership.

These are the logs in node2:
core: parsing information for new active node: active_cluster_addr=https://node3.testing.net:8201 active_redirect_addr=https://node3.testing.net:8200
core.cluster-listener: creating rpc dialer: address=node3.testing.net:8201 alpn=req_fw_sb-act_v1 host=fw-adc20532-e019-043e-ebc4-0c26f37077ab
core: forwarding: error sending echo request to active node: error="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: remote error: tls: internal error""

This is mysterious, because I can open ssl connection to node3 using the configured tls cert and key in openbao configuration, so not sure what exactly is the tls error.

[DrDaveD]

I remember struggling with similar problems when I was setting up Vault raft. I don't remember how exactly I debugged it, but my first thought is to make sure you have have the CA cert properly available. I use leader_ca_cert_file in the raft configuration.