Kafka pods in a failed state as it is unable to get the kraft quorum state

[shreyasarani23]

Below is my kafknodepool spec

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: controller
  namespace: observability
  labels:
    strimzi.io/cluster: kraft-kafka-cluster
spec:
  replicas: 3
  roles:
    - controller
  storage:
    type: ephemeral
  resources: 
    requests:
      memory: 4Gi
      cpu: "201m"
    limits:
      memory: 8Gi
      cpu: "2"
  template:
    pod:
      metadata:
        labels:
          app: kafka-controller
        annotations:
          prometheus.io/scrape: 'true'
          prometheus.io/path: '/metrics'
          prometheus.io/port: '9404'
          prometheus.io/scheme: 'http'
      affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: dedicated-for
                      operator: In
                      values:
                        - kafka-spot
      tolerations:
        - key: dedicated-for
          value: kafka-spot
          effect: "NoExecute"
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: broker
  namespace: observability
  labels:
    strimzi.io/cluster: kraft-kafka-cluster
spec:
  replicas: 3
  roles:
    - broker
  storage:
    type: persistent-claim
    size: 42Gi
    class: kafka-sc
    deleteClaim: true
  resources: 
    requests:
      memory: 4Gi
      cpu: "200m"
    limits:
      memory: 8Gi
      cpu: "2"
  jvmOptions:
    -Xms: 4096m
    -Xmx: 8192m
  template:
    pod:
      metadata:
        labels:
          app: kafka-broker
        annotations:
          prometheus.io/scrape: 'true'
          prometheus.io/path: '/metrics'
          prometheus.io/port: '9404'
          prometheus.io/scheme: 'http'
      affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: dedicated-for
                      operator: In
                      values:
                        - kafka-spot
      tolerations:
        - key: dedicated-for
          value: kafka-spot
          effect: "NoExecute"
---

And here is my kafka spec

---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kraft-kafka-cluster
  namespace: observability
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:
  kafka:
    version: 3.9.0
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    metadataVersion: 3.9-IV0
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 2
      min.insync.replicas: 1
      log.retention.ms: 86400000
      auto.create.topics.enable: "false"
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics.yaml
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: dedicated-for
                      operator: In
                      values:
                        - kafka-spot
        tolerations:
          - key: dedicated-for
            value: kafka-spot
            effect: "NoExecute"
  entityOperator:
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: dedicated-for
                      operator: In
                      values:
                        - kafka-spot
        tolerations:
          - key: dedicated-for
            value: kafka-spot
            effect: "NoExecute"
    topicOperator: {}
    userOperator: {}
  cruiseControl:
    autoRebalance:
      - mode: add-brokers
        template:
          name: custom-rebalance-template
      - mode: remove-brokers
        template:
          name: custom-rebalance-template
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: cruise-control-metrics
          key: metrics-config.yml
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: dedicated-for
                      operator: In
                      values:
                        - kafka-spot
        tolerations:
          - key: dedicated-for
            value: kafka-spot
            effect: "NoExecute"
---

With the above spec the kafka cluster was running fine but suddenly yesterday we are encountering this issue
in all of the broker pods.

2025-02-24 08:11:29,569 ERROR Encountered fatal fault: Unexpected error in raft IO thread (org.apache.kafka.server.fault.ProcessTerminatingFaultHandler) [kafka-2-raft-io-thread]
java.lang.IllegalArgumentException: Attempt to truncate to offset 0, which is below the current high watermark 433033
	at kafka.raft.KafkaMetadataLog.truncateTo(KafkaMetadataLog.scala:170)
	at org.apache.kafka.raft.ReplicatedLog.truncateToEndOffset(ReplicatedLog.java:227)
	at kafka.raft.KafkaMetadataLog.truncateToEndOffset(KafkaMetadataLog.scala:52)
	at org.apache.kafka.raft.KafkaRaftClient.handleFetchResponse(KafkaRaftClient.java:1624)
	at org.apache.kafka.raft.KafkaRaftClient.handleResponse(KafkaRaftClient.java:2457)
	at org.apache.kafka.raft.KafkaRaftClient.handleInboundMessage(KafkaRaftClient.java:2613)
	at org.apache.kafka.raft.KafkaRaftClient.poll(KafkaRaftClient.java:3312)
	at org.apache.kafka.raft.KafkaRaftClientDriver.doWork(KafkaRaftClientDriver.java:64)
	at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:136)

2025-02-24 08:23:32,091 ERROR Encountered fatal fault: Unexpected error in raft IO thread (org.apache.kafka.server.fault.ProcessTerminatingFaultHandler) [kafka-1-raft-io-thread]
java.lang.IllegalArgumentException: Attempt to truncate to offset 0, which is below the current high watermark 433267
	at kafka.raft.KafkaMetadataLog.truncateTo(KafkaMetadataLog.scala:170)
	at org.apache.kafka.raft.ReplicatedLog.truncateToEndOffset(ReplicatedLog.java:227)
	at kafka.raft.KafkaMetadataLog.truncateToEndOffset(KafkaMetadataLog.scala:52)
	at org.apache.kafka.raft.KafkaRaftClient.handleFetchResponse(KafkaRaftClient.java:1624)
	at org.apache.kafka.raft.KafkaRaftClient.handleResponse(KafkaRaftClient.java:2457)
	at org.apache.kafka.raft.KafkaRaftClient.handleInboundMessage(KafkaRaftClient.java:2613)
	at org.apache.kafka.raft.KafkaRaftClient.poll(KafkaRaftClient.java:3312)
	at org.apache.kafka.raft.KafkaRaftClientDriver.doWork(KafkaRaftClientDriver.java:64)
	at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:136)

2025-02-24 08:20:02,674 ERROR Encountered fatal fault: Unexpected error in raft IO thread (org.apache.kafka.server.fault.ProcessTerminatingFaultHandler) [kafka-0-raft-io-thread]
java.lang.IllegalArgumentException: Attempt to truncate to offset 0, which is below the current high watermark 439601
	at kafka.raft.KafkaMetadataLog.truncateTo(KafkaMetadataLog.scala:170)
	at org.apache.kafka.raft.ReplicatedLog.truncateToEndOffset(ReplicatedLog.java:227)
	at kafka.raft.KafkaMetadataLog.truncateToEndOffset(KafkaMetadataLog.scala:52)
	at org.apache.kafka.raft.KafkaRaftClient.handleFetchResponse(KafkaRaftClient.java:1624)
	at org.apache.kafka.raft.KafkaRaftClient.handleResponse(KafkaRaftClient.java:2457)
	at org.apache.kafka.raft.KafkaRaftClient.handleInboundMessage(KafkaRaftClient.java:2613)
	at org.apache.kafka.raft.KafkaRaftClient.poll(KafkaRaftClient.java:3312)
	at org.apache.kafka.raft.KafkaRaftClientDriver.doWork(KafkaRaftClientDriver.java:64)
	at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:136)

But the controller pods are running fine. Even after trying to restart the kafka pods we are facing the same issue. Any idea on this? I guess it has something to do with __cluster_metadata and kraft quorum. Let me know if you need more info on this

Strimzi Version: 0.45.0 (installed through helm chart)
Environment: EKS k8's 1.31

[scholzj]

Maybe that is related to the ephemeral storage used in your controllers that can be lost anytime? That would essentially truncate the metadata log.

[shreyasarani23]

Ohk you mean to say that since the controller pods are configured with ephemeral storage if the node where these controller pods were running gets drained/removed for some reason, and brings up a new node and these controller pods gets scheduled on this new node the cluster metadata will be lost so the the broker is trying to truncate the offsets ? or is there a interval/property configured at the broker level to truncate the old offsets for the metadata log?

Also how do I fix this ? Since I was doing some POC I didn't configure persistence for controller pods. Configuring persistence to controller pods is the only way to fix this or is there any other alternative way?

[scholzj]

I'm not a Kafka expert. But the way I read it is that you lost all storage on the controllers, so they start from 0. But the brokers know that they had it at offset 439601 and are confused / out of sync.

I don't know how you can fix it -> I think you should delete the cluster and start a fresh one with proper storage. I'm not sure if / how you can recover the existing cluster without loosing the data.