Intermittent connection errors to the broker

[alok87]

Error while closing connection to broker internal-ELB-address 9094: write tcp 100.102.123.15:35898->10.2.20.119:9094: write: broken pipe

Keep getting these errors.

I have a kafka setup with loadbalancers for the brokers as explained in this article https://strimzi.io/blog/2019/05/13/accessing-kafka-part-4/

The broker does not have any errors. But I keep getting this.

Please suggest how to debug this to find what could be wrong in the setup.

[scholzj]

So is the Kafka cluster running somewhere else? In other Kube cluster? Or why do you use the load balancers?

[scholzj]

You can configure multiple listeners in the Kafka cluster - so you should be able to add a new internal one and give it a try (it will roll the Kafka cluster, but if you use replication you should be fine)

[alok87]

thanks i will give this a try.

[alok87]

hey! just an update: after moving to internal communication, the errors have disappeared. But the topics being processed also reduced, so not 100% sure, will get back.

Just curious why the ELB is a problem. I see in the ELB the healthy nodes fluctuates mostly due to spot ones coming and going. Could this be the reason? Any recommendation to avoid it?

For us data transfer rate is not a problem so much, as it is in the same AZ. Keeping the kafka and consumers run separately is important to us as developers keep switching clusters.

How to be sure...

[scholzj]

For us data transfer rate is not a problem so much, as it is in the same AZ. Keeping the kafka and consumers run separately is important to us as developers keep switching clusters.

Well, you still pay for the ELB transfers, or? Even if it would be the same AZ. But that was really just a comment ... that is your call to make.

But the topics being processed also reduced, so not 100% sure, will get back.

Sorry, but not sure what does this mean. You mean that you had other changes in your environment so maybe it is coincidence?

Just curious why the ELB is a problem. I see in the ELB the healthy nodes fluctuates mostly due to spot ones coming and going. Could this be the reason? Any recommendation to avoid it?

Could be. Kafka clients use long running connections. As far as I understand it ... normally, ELB forwards the connection to one of the worker nodes behind the ELB and then kube-proxy forwards it to the right node where Kafka is running. And If the worker node in the middle is shutdown, the connection will be terminated. So if the workers in your cluster come and go often, I guess the connections might be terminated just as they shutdown in case there were used to forward the connection. Maybe the External Traffic Policy can help you (https://strimzi.io/docs/operators/latest/full/using.html#property-listener-config-traffic-policy-reference). That should in theory steer the connections to connect directly to the node with the Kafka broker and not through the other workers. But not 100% sure about it - not sure how will will the ELB take it ... but you can give it a try.

[alok87]

thanks a lot for the replies scholz.