Upgrade Kafka client version to 3.7.0

[tomstepp]

Upgrade Kafka client version to 3.7.0

---

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[tomstepp]

R: @Abacn

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

[Abacn]

from my understanding currently Beam compiles on older Kafka version but users are supposed to provide their own version of kafka-client:

beam/sdks/java/io/kafka/build.gradle

Line 62 in 332a61d

provided library.java.kafka_clients

. it is marked as a "provided" dependency so that user need to add kafka-client dependency in their pipeline

For example, in io-expansion service (used as a standalone jar) we indeed used a newer kafka-client:

beam/sdks/java/io/expansion-service/build.gradle

Line 33 in ebef816

resolutionStrategy.force 'org.apache.kafka:kafka-clients:3.1.2'

[tomstepp]

from my understanding currently Beam compiles on older Kafka version but users are supposed to provide their own version of kafka-client:

beam/sdks/java/io/kafka/build.gradle

Line 62 in 332a61d

provided library.java.kafka_clients

. it is marked as a "provided" dependency so that user need to add kafka-client dependency in their pipeline
For example, in io-expansion service (used as a standalone jar) we indeed used a newer kafka-client:

beam/sdks/java/io/expansion-service/build.gradle

Line 33 in ebef816

resolutionStrategy.force 'org.apache.kafka:kafka-clients:3.1.2'

I see, thanks! I am okay with just upgrading the internal test version, I think that will work. However, I wonder if Beam should upgrade to a newer version for test purposes as well?

Any thoughts @Abacn, @johnjcasey @an2x?