Issue with Standalone User Operator: "Cannot extract trust set from null secret" when TLS is disabled

[iMaxZxC]

Hello,

I'm encountering an issue with the standalone User Operator (version 0.45.0) when attempting to manage users on a remote Kafka cluster (deployed on virtual machines, separate from my Kubernetes cluster). My setup does not use TLS, and I use SCRAM-SHA-512 for authentication with the credentials:

username: alice
password: alice-secret
I have set the following environment variables in my User Operator deployment:

Exception in thread "main" java.lang.NullPointerException: Cannot extract trust set from null secret.
    at java.base/java.util.Objects.requireNonNull(Objects.java:235)
    at io.strimzi.operator.common.auth.PemTrustSet.<init>(PemTrustSet.java:41)
    at io.strimzi.operator.user.Main.createAdminClient(Main.java:147)
    at io.strimzi.operator.user.Main.main(Main.java:82)

Configuration Details:
My User Operator deployment (in namespace kafka) is configured as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-kraft-test-strimzi-user-operator
  namespace: kafka
  labels:
    app: strimzi
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-kraft-test-strimzi-user-operator
  template:
    metadata:
      labels:
        app: kafka-kraft-test-strimzi-user-operator
    spec:
      serviceAccountName: kafka-kraft-test-strimzi-user-operator
      containers:
        - name: kafka-kraft-test-strimzi-user-operator
          image: quay.io/strimzi/operator:0.45.0
          command: ["/opt/strimzi/bin/user_operator_run.sh"]
          env:
            - name: STRIMZI_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: STRIMZI_KAFKA_BOOTSTRAP_SERVERS
              value: "example-broker1:9093,example-broker2:9093,example-broker3:9093"
            - name: STRIMZI_LABELS
              value: "strimzi.io/cluster=kafka-kraft-test"
            - name: STRIMZI_FULL_RECONCILIATION_INTERVAL_MS
              value: "120000"
            - name: STRIMZI_CA_CERT_NAME
              value: "unused"
            - name: STRIMZI_CA_KEY_NAME
              value: "unused"
            - name: STRIMZI_CA_VALIDITY
              value: "365"
            - name: STRIMZI_CA_RENEWAL
              value: "30"
            - name: STRIMZI_WORK_QUEUE_SIZE
              value: "10000"
            - name: STRIMZI_CONTROLLER_THREAD_POOL_SIZE
              value: "10"
            - name: STRIMZI_USER_OPERATIONS_THREAD_POOL_SIZE
              value: "4"
            - name: STRIMZI_LOG_LEVEL
              value: "INFO"
            - name: STRIMZI_GC_LOG_ENABLED
              value: "true"
            - name: STRIMZI_SECRET_PREFIX
              value: "kafka-"
            - name: STRIMZI_ACLS_ADMIN_API_SUPPORTED
              value: "true"
            - name: STRIMZI_KAFKA_ADMIN_CLIENT_CONFIGURATION
              value: |
                default.api.timeout.ms=120000
                request.timeout.ms=60000
            - name: STRIMZI_TLS_ENABLED
              value: "false"
            - name: STRIMZI_SASL_ENABLED
              value: "true"
            # Using JAAS config only – not mixing with individual SASL variables:
            - name: STRIMZI_SASL_JAAS_CONFIG
              value: "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"alice\" password=\"alice-secret\";"
            - name: STRIMZI_SECURITY_PROTOCOL
              value: "SASL_PLAINTEXT"
            - name: STRIMZI_TLS_AUTH_ENABLED
              value: "false"

Since I'm not using TLS, I've created a dummy secret named unused in the kafka namespace:

apiVersion: v1
kind: Secret
metadata:
  name: unused
  namespace: kafka
type: Opaque
data:
  ca.crt: ZHVtbXk= 
  ca.key: ZHVtbXk=

JAAS Configuration: Is my JAAS configuration correct for SCRAM‑SHA‑512? (I'm using the single variable STRIMZI_SASL_JAAS_CONFIG with the value:
org.apache.kafka.common.security.scram.ScramLoginModule required username="alice" password="alice-secret";
Is that the recommended approach?)

TLS Requirement: Given that TLS is disabled (STRIMZI_TLS_ENABLED is "false"), is it expected that the User Operator still requires valid CA secrets? Is using a dummy secret like the one above a valid workaround?

Any Additional Configuration: Are there any other changes or workarounds required to run the User Operator successfully in a non-TLS environment with SCRAM‑SHA‑512?

[scholzj]

You should probably start by sharing the full log.

[scholzj]

ALso, keep in mind that there is nothing like STRIMZI_TLS_ENABLED or STRIMZI_TLS_AUTH_ENABLED in the User Operator. Nor do the SASL environment variables.

[iMaxZxC]

	namespace='kafka'
	reconciliationIntervalMs=120000
	kafkaBootstrapServers='example-broker1:9093,example-broker2:9093,example-broker3:9093'
	labels=`Labels{strimzi.io/cluster=kafka-kraft-test}'
	caCertSecretName='unused'
	caKeySecretName='unused'
	clusterCaCertSecretName='null'
	euoKeySecretName='null'
	caNamespace='kafka'
	secretPrefix='kafka-'
	clientsCaValidityDays=365
	clientsCaRenewalDays=30
	aclsAdminApiSupported=true
	scramPasswordLength=32
	maintenanceWindows=`null'
	kafkaAdminClientConfiguration=`{request.timeout.ms=60000, default.api.timeout.ms=120000}'
	operationTimeoutMs=300000
	workQueueSize=10000
	controllerThreadPoolSize=10
	cacheRefresh=15000
	batchQueueSize=1024
	batchMaxBlockSize=100
	batchMaxBlockTime=100
	userOperationsThreadPoolSize=4
	featureGates='FeatureGates(ContinueReconciliationOnManualRollingUpdateFailure=true)'}[
2025-02-28T11:03:07.491+0000] GC(14) Pause Young (Allocation Failure)
[2025-02-28T11:03:07.493+0000] GC(14) DefNew: 4204K(4352K)->132K(4352K) Eden: 3890K(3904K)->0K(3904K) From: 314K(448K)->132K(448K)
[2025-02-28T11:03:07.493+0000] GC(14) Tenured: 5696K(9496K)->5971K(9496K)
[2025-02-28T11:03:07.493+0000] GC(14) Metaspace: 10632K(10880K)->10632K(10880K) NonClass: 9415K(9536K)->9415K(9536K) Class: 1217K(1344K)->1217K(1344K)
[2025-02-28T11:03:07.493+0000] GC(14) Pause Young (Allocation Failure) 9M->5M(13M) 1.526ms
[2025-02-28T11:03:07.493+0000] GC(14) User=0.00s Sys=0.00s Real=0.01s
[2025-02-28T11:03:07.598+0000] GC(15) Pause Young (Allocation Failure)
[2025-02-28T11:03:07.599+0000] GC(15) DefNew: 4030K(4352K)->223K(4352K) Eden: 3898K(3904K)->0K(3904K) From: 132K(448K)->223K(448K)
[2025-02-28T11:03:07.599+0000] GC(15) Tenured: 5971K(9496K)->5971K(9496K)
[2025-02-28T11:03:07.599+0000] GC(15) Metaspace: 11743K(11968K)->11743K(11968K) NonClass: 10384K(10496K)->10384K(10496K) Class: 1359K(1472K)->1359K(1472K)
[2025-02-28T11:03:07.599+0000] GC(15) Pause Young (Allocation Failure) 9M->6M(13M) 0.944ms
[2025-02-28T11:03:07.599+0000] GC(15) User=0.00s Sys=0.00s Real=0.00s
[2025-02-28T11:03:07.784+0000] GC(16) Pause Young (Allocation Failure)
[2025-02-28T11:03:07.785+0000] GC(16) DefNew: 4120K(4352K)->429K(4352K) Eden: 3896K(3904K)->0K(3904K) From: 223K(448K)->429K(448K)
[2025-02-28T11:03:07.785+0000] GC(16) Tenured: 5971K(9496K)->5971K(9496K)
[2025-02-28T11:03:07.785+0000] GC(16) Metaspace: 12871K(13120K)->12871K(13120K) NonClass: 11363K(11456K)->11363K(11456K) Class: 1507K(1664K)->1507K(1664K)
[2025-02-28T11:03:07.785+0000] GC(16) Pause Young (Allocation Failure) 9M->6M(13M) 1.698ms
[2025-02-28T11:03:07.785+0000] GC(16) User=0.00s Sys=0.00s Real=0.00s
[2025-02-28T11:03:08.084+0000] GC(17) Pause Young (Allocation Failure)
[2025-02-28T11:03:08.088+0000] GC(17) DefNew: 4333K(4352K)->448K(4352K) Eden: 3904K(3904K)->0K(3904K) From: 429K(448K)->448K(448K)
[2025-02-28T11:03:08.088+0000] GC(17) Tenured: 5971K(9496K)->7004K(9496K)
[2025-02-28T11:03:08.088+0000] GC(17) Metaspace: 14315K(14592K)->14315K(14592K) NonClass: 12653K(12800K)->12653K(12800K) Class: 1661K(1792K)->1661K(1792K)
[2025-02-28T11:03:08.088+0000] GC(17) Pause Young (Allocation Failure) 10M->7M(13M) 3.717ms
[2025-02-28T11:03:08.088+0000] GC(17) User=0.01s Sys=0.00s Real=0.00s
[2025-02-28T11:03:08.380+0000] GC(18) Pause Young (Allocation Failure)
[2025-02-28T11:03:08.384+0000] GC(18) DefNew: 4352K(4352K)->448K(4352K) Eden: 3904K(3904K)->0K(3904K) From: 448K(448K)->448K(448K)
[2025-02-28T11:03:08.384+0000] GC(18) Tenured: 7004K(9496K)->7753K(9496K)
[2025-02-28T11:03:08.384+0000] GC(18) Metaspace: 15421K(15680K)->15421K(15680K) NonClass: 13649K(13760K)->13649K(13760K) Class: 1771K(1920K)->1771K(1920K)
[2025-02-28T11:03:08.384+0000] GC(18) Pause Young (Allocation Failure) 11M->8M(13M) 3.992ms
[2025-02-28T11:03:08.384+0000] GC(18) User=0.00s Sys=0.00s Real=0.01s
[2025-02-28T11:03:08.680+0000] GC(19) Pause Young (Allocation Failure)
[2025-02-28T11:03:08.683+0000] GC(19) DefNew: 4352K(4352K)->363K(4352K) Eden: 3904K(3904K)->0K(3904K) From: 448K(448K)->363K(448K)
[2025-02-28T11:03:08.684+0000] GC(19) Tenured: 7753K(9496K)->8197K(9496K)
[2025-02-28T11:03:08.684+0000] GC(19) Metaspace: 17496K(17792K)->17496K(17792K) NonClass: 15404K(15552K)->15404K(15552K) Class: 2092K(2240K)->2092K(2240K)
[2025-02-28T11:03:08.684+0000] GC(19) Pause Young (Allocation Failure) 11M->8M(13M) 3.190ms
[2025-02-28T11:03:08.684+0000] GC(19) User=0.00s Sys=0.01s Real=0.01s
Exception in thread "main" java.lang.NullPointerException: Cannot extract trust set from null secret.
	at java.base/java.util.Objects.requireNonNull(Objects.java:235)
	at io.strimzi.operator.common.auth.PemTrustSet.<init>(PemTrustSet.java:41)
	at io.strimzi.operator.user.Main.createAdminClient(Main.java:147)
	at io.strimzi.operator.user.Main.main(Main.java:82)
[2025-02-28T11:03:08.886+0000] Heap
[2025-02-28T11:03:08.886+0000]  def new generation   total 4352K, used 3610K [0x00000000fd800000, 0x00000000fdcb0000, 0x00000000fe550000)
[2025-02-28T11:03:08.886+0000]   eden space 3904K,  83% used [0x00000000fd800000, 0x00000000fdb2b8f0, 0x00000000fdbd0000)
[2025-02-28T11:03:08.886+0000]   from space 448K,  81% used [0x00000000fdc40000, 0x00000000fdc9af28, 0x00000000fdcb0000)
[2025-02-28T11:03:08.886+0000]   to   space 448K,   0% used [0x00000000fdbd0000, 0x00000000fdbd0000, 0x00000000fdc40000)
[2025-02-28T11:03:08.886+0000]  tenured generation   total 9496K, used 8197K [0x00000000fe550000, 0x00000000fee96000, 0x0000000100000000)
[2025-02-28T11:03:08.886+0000]    the space 9496K,  86% used [0x00000000fe550000, 0x00000000fed514b0, 0x00000000fed51600, 0x00000000fee96000)
[2025-02-28T11:03:08.886+0000]  Metaspace       used 18527K, committed 18880K, reserved 1114112K
[2025-02-28T11:03:08.886+0000]   class space    used 2282K, committed 2432K, reserved 1048576K

[scholzj]

Do you have the log from the beginning when it starts? That way it will show the full configuration etc. If needed, you can disable the GC logs using STRIMZI_GC_LOG_ENABLED to get rid of that as it is unrelated anyway.

[iMaxZxC]

full log in attachments, the thing is that in my kafka namespace 2 more kafka clusters are deployed, and in the log they are displayed but I don't know the reason for it.
kafka-kraft-test-strimzi-user-operator-b5965bb96.log