Using drain-cleaner with local persistent storage

[applejag]

Is it possible to use strimzi-drain-cleaner with a StorageClass that locks PVs/PVCs to specific nodes?

We're running Kubernetes on bare metal and are using OpenEBS LVM LocalPV storage provider (https://github.com/openebs/lvm-localpv, which is basically just hostpath storage with extra steps) to squeeze the most performance out of our drives.

When drain-cleaner marks the pod with the annotation, the operator kills that pod, and then because the node is cordoned the new pod is stuck in Pending.

This is quite fine from my perspective as then the pod at least got a clean shutdown.

But this still prevents the drain from succeeding.

We have 3 kafka brokers and 3 zookeepers. When we drain, only 1 zookeeper or only 1 broker gets terminated. But then I guess the cluster is not restored yet so the Strimzi cluster operator won't terminate the other pod.

Is there a configuration where I can tell Strimzi that having 1 node down is OK, and that it should continue with the manual restarts? And then also, is it possible to tell drain-cleaner to not invalidate the webhook requests when the pods are just stuck in pending?

[scholzj]

It should work as long as you have only one Kafka broker (from the same Kafka cluster) on given node and as long as you drain the nodes one by one.

[applejag]

Something I failed to emphasize was: Strimzi only restarts EITHER 1 Zookeeper pod OR 1 Kafka broker pod.

To give an overview, these are our pods:

NAME                  READY   STATUS    NODE
strimzi-kafka-0       1/1     Running   node0
strimzi-kafka-1       1/1     Running   node1
strimzi-kafka-2       1/1     Running   node2
strimzi-zookeeper-0   1/1     Running   node0
strimzi-zookeeper-1   1/1     Running   node1
strimzi-zookeeper-2   1/1     Running   node2

So step by step, this is what happens:

1. kubectl drain node0

2. eviction event rejected on strimzi-zookeeper-0 by strimzi-drain-cleaner

3. eviction event rejected on strimzi-kafka-0 by strimzi-drain-cleaner

4. strimzi-drain-cleaner annotates strimzi-zookeeper-0 with strimzi.io/manual-rolling-update=true

5. strimzi-drain-cleaner annotates strimzi-kafka-0 with strimzi.io/manual-rolling-update=true

6. strimzi operator drains and restarts strimzi-zookeeper-0, and the pods now look like this:

   NAME                  READY   STATUS    NODE
   strimzi-kafka-0       1/1     Running   node0
   strimzi-kafka-1       1/1     Running   node1
   strimzi-kafka-2       1/1     Running   node2
   strimzi-zookeeper-0   0/1     Pending   <none>
   strimzi-zookeeper-1   1/1     Running   node1
   strimzi-zookeeper-2   1/1     Running   node2

7. nothing else happens, it is stuck, and kubectl drain node0 will never succeed

In this state the Strimzi operator seems to be waiting for strimzi-zookeeper-0 before continuing on with draining and restarting strimzi-kafka-0. But as our PVCs are locked to the nodes, then strimzi-zookeeper-0 will never come back online until we uncodron node0.

[scholzj]

I guess you could move to KRaft to get rid of the ZooKeeper or you can just disable the Drain Cleaner for ZooKeeper and have Kubernetes evict it based on PDB with maxUnavailable 1. The ZooKeper sync the data differently, so there should not be an issue with that.

[applejag]

Ah yeah. And seeing as Zookeeper is on its way out the door and will be unsupported in the next Strimzi release, then that's the path forward anyways.

Thanks for answering minutes apart even though I answered months apart :D Impressive!

[scholzj]

That said ... you would have the same issue with 2 Kafka nodes on a single node. So KRaft can help only with mixed nodes. Having a controller-only and broker-only nodes on the same wrker node with local storage would cause the same issue.

[applejag]

Yeah some good pod anti affinity takes care of that. We do not co-locate any brokers on the same nodes. And if we did, we wouldn't expect things like strimzi-drain-cleaner to work.

[applejag]

I just tried disabling the drain cleaner on zookeeper pods via the STRIMZI_DRAIN_ZOOKEEPER env var, but that didn't help either.

So the only solutions I guess would be to switch to KRaft or to not co-locate anything.

I also found your response over at https://github.com/orgs/strimzi/discussions/9932#discussioncomment-9049391 where the same conclusions were taken. Basically: use more nodes, or move to KRaft.