Ciphers change causes brokers failure

[lenin-joseph]

Hi,

Whenever we change the ciphers, the new ciphers are being applied to all the brokers dynamically. However, if we restart brokers after applying the ciphers, brokers go to crashloopbackoff and throw the below errors.
So, I would like to understand if this is a bug/limitation or an expected behavior. OR should this be checked with apache-kafka community?

strimzi-kafka-operator-0.44.0

Steps to reproduce:
Deploy a kafka cluster with the below CR,
Once the cluster is up, change the cipher to TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
Once the operator reconciles, restart a broker and check logs.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: my-cluster
spec:
kafka:
# ...
config:
ssl.cipher.suites: TLS_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256

Errors in broker logs:

2024-11-16 07:53:06,113 ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) [main]
org.apache.kafka.common.config.ConfigException: Invalid value javax.net.ssl.SSLHandshakeException: no cipher suites in common for configuration A client SSLEngine created with the provided settings can't connect to a server SSLEngine created with those settings.
at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:103)
at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:73)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192)
at org.apache.kafka.common.network.ChannelBuilders.serverChannelBuilder(ChannelBuilders.java:107)
at kafka.network.Processor.(SocketServer.scala:973)
at kafka.network.Acceptor.newProcessor(SocketServer.scala:879)
at kafka.network.Acceptor.$anonfun$addProcessors$1(SocketServer.scala:849)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:190)
at kafka.network.Acceptor.addProcessors(SocketServer.scala:848)
at kafka.network.DataPlaneAcceptor.configure(SocketServer.scala:523)
at kafka.network.SocketServer.createDataPlaneAcceptorAndProcessors(SocketServer.scala:251)
at kafka.network.SocketServer.$anonfun$new$31(SocketServer.scala:175)
at kafka.network.SocketServer.$anonfun$new$31$adapted(SocketServer.scala:175)
at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576)
at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574)
at scala.collection.AbstractIterable.foreach(Iterable.scala:933)
at kafka.network.SocketServer.(SocketServer.scala:175)
at kafka.server.KafkaServer.startup(KafkaServer.scala:380)
at kafka.Kafka$.main(Kafka.scala:112)
at kafka.Kafka.main(Kafka.scala)

[scholzj]

I guess it suggests the ciphers you are trying to set are not suported?

[lenin-joseph]

Thanks for your response.

Whenever we use any unsupported ciphers(ex: TLS_ECDHE_PSK_WITH_AES_128_CCM_SHA256), Kafka throws "unsupported" exceptions. But not when I use the entire list below, which means all the ciphers in the below list are supported.

If I try with only TLS_ECDHE_ECDSA*, it throws "no cipher suites in common". So, it looks like TLS_ECDHE_ECDSA* doesn't work with certain certificate types.

TLS_AES_128_GCM_SHA256
TLS_AES_256_GCM_SHA384
TLS_CHACHA20_POLY1305_SHA256
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

[scholzj]

I think there is a difference between unknown cipher suites and those that are known but not supported. But I might be wrong.

[scholzj]

And yes, some cipher suites might need some types of certificates.