Kafka Brokers not able to come back up after a node reset/flip

[ssandanshi]

Describe the bug
We have a Kafka Cluster running without any issues. Somehow the node(s) it is running on all restart. After the node restart, the zookeeper comes back up but the Kafka brokers remain in a crashloopbackoff.

We had seen the issue previously and started a discussion thread. That incident happened on an AWS EKS( v1.18.9-eks) hosted cluster. The current issue happened on a docker desktop(v 2.5.0.1) (running on Mac) hosted Kubernetes environment (v1.19.3)

To Reproduce
Steps to reproduce the behavior:

1. Install the Cluster Operator using default helm chart.
2. Install a kafka cluster resource
3. After kafka is up, Disable kubernetes in docker desktop UI and click apply and restart. This will stop the docker desktop node
   
4. After Kubernetes is disabled, Enable Kubernetes again by clicking to enable kubernetes in Docker desktop. This will start the docker desktop node
5. When the kubernetes cluster comes back up, The Kafka CR would be there. The zookeeper pods are in running but kafka brokers are in a crashloop back off

Expected behavior
Kafka cluster should be running after a node restart

Environment (please complete the following information):

• Strimzi version: 0.20.0
• Installation method: Helm charts (we use argoCD to deploy helm charts But it should be reproducible using helm charts)
• Kubernetes cluster: v1.19.3 (docker desktop hosted)
• Infrastructure: Docker Desktop(v 2.5.0.1) (running on Mac) hosted Kubernetes environment (v1.19.3) and EKS earlier(see discussion tagged)

YAML files and logs
report.sh assumes that the cluster CR and namespace reside in the same namespace. That is not the case for us. I updated the report.sh script to accept namespace for cluster operator separately.
report.sh.zip

reports.zip
kafka namespace - 2269-streaming
Cluster operator namespace - 2269-operations

In order to overcome docker rate limits, We use AWS ECR to host our images. I manually replaced all ECR images in yaml manifests and logs with their equivalent docker images in the report folder. Let me know if this causes any issues.

Additional context
Previous discussion post - #4767

[scholzj]

TBH, I'm not sure what the disabling and enabling of the Kubernetes in Docker Desktop really does. For example, does the persistent storage survive this without any issues etc.? From the logs, there seems to be issue with your Zookeeper but it is not really clear what caused it. Have you tried to restart the Zookeeper cluster?

[ssandanshi]

More details on what disable and enable kubernetes does - https://docs.docker.com/desktop/kubernetes/

Judging by the age of the kafka PVC(s), they do survive this. historically, I've seen old PVCs on EKS reattach after recreating the Kafka CR.

I was able to exec into the zookeeper pod and run the zookeeper-shell script. Zookeeper pods are running. Kafka and the topic, user operators are unable to connect to ZK. Could you expand by what you mean by restart of zookeeper cluster, Should I delete one or more pods or modify the stateful set or anything along that line?

[scholzj]

Judging by the age of the kafka PVC(s), they do survive this. historically, I've seen old PVCs on EKS reattach after recreating the Kafka CR.

But EKS is not the same as the Docker Desktop. Docker Desktop does not have real persistent volumes I guess, just directories somewhere on the disk as other similar distributions. So the PVs might exist, but the files might be gone.

I was able to exec into the zookeeper pod and run the zookeeper-shell script. Zookeeper pods are running. Kafka and the topic, user operators are unable to connect to ZK. Could you expand by what you mean by restart of zookeeper cluster, Should I delete one or more pods or modify the stateful set or anything along that line?

Well, if you run Zookeeper shell, can you list some date from Zookeeper? Can you list the configuration? If not, the cluster does not have a quorum.

With restarting the cluster I really just mean deleting the pods.