Connection leakage causes performance degradation in the Kafka connector

[sajith-madhusanka]

Description

While conducting a performance evaluation on the Kafka connector message publishing operation, it was observed that the MI server became unresponsive, and the performance was significantly low. The following two issues were identified during further debugging:

1. While connections to the Kafka broker were being properly closed, two new connections (for retrieving the key schema and value schema) were created for every request. These connections were not properly returned to the connection pool.
2. A large number of abandoned connections were created, while only one connection was actively used for message publishing.

Steps to Reproduce

• Deploy a proxy service with a Kafka publish operation:

  <kafkaTransport.publishMessages configKey="KAFKA_CONNECT">
      <topic>{$ctx:kafka_topic}</topic>
      <keySchemaId>1</keySchemaId>
      <valueSchemaId>1</valueSchemaId>
  </kafkaTransport.publishMessages>

• The Kafka connection should be configured with connection pooling enabled:

  <kafkaTransport.init>
      <connectionType>kafka</connectionType>
      <name>KAFKA_CONNECT</name>
      - - - - - - - - - 
      - - - - - - - - - 
      <poolingEnabled>true</poolingEnabled>
  </kafkaTransport.init>

• Perform a load test with 20 concurrent users.

  • You will observe that the MI server becomes unresponsive due to exhausting the PassThru Message Processor threads.

Version

3.3.1

Environment Details (with versions)

No response