[draft] [testing] update kafka version used in testing to 3.6.0

[khushijain21]

Proposed commit message

Updated kafka version used in our integration tests

Error logs

Error while executing ACL command: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: createAcls
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: createAcls
        at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
        at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
        at kafka.admin.AclCommand$AdminClientService.$anonfun$addAcls$3(AclCommand.scala:115)
        at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:985)
        at scala.collection.immutable.Map$Map3.foreach(Map.scala:376)
        at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:984)
        at kafka.admin.AclCommand$AdminClientService.$anonfun$addAcls$1(AclCommand.scala:112)
        at kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:111)
        at kafka.admin.AclCommand$.main(AclCommand.scala:73)
        at kafka.admin.AclCommand.main(AclCommand.scala)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: createAcls

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Kafka version 4.0 will not be compatible with client libraries using protocol version less than 2.1.0

Related issues

Closes #41654

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @khushijain21? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[khushijain21]

@elastic/obs-infraobs-integrations can someone from your team take a look. I've upgraded kafka to 3.6.0 and made corresponding changes according to docs. The broker seems to not be either listening on localhost 9091 or is inaccessible. The error logs are attached in description.

cc: @lalit-satapathy

[rdner]

@khushijain21 please have a look at the CI failures, it's Kafka related and looks relevant to the changes.

[jlind23]

@andresrc can we please have someone looking at this please? Wrt #41833 (comment)

[lalit-satapathy]

@andresrc can we please have someone looking at this please? Wrt #41833 (comment)

@Linu-Elias is taking a look.

[Linu-Elias]

Please check the CI failures, they are mostly because the broker seems to not be listening on localhost:9091

[Linu-Elias]

I would suggest uncommenting this, and we can still wait for the health check to complete.

[lalit-satapathy]

Hi @khushijain21,

Can you summarise what is exactly pending on this PR? Is there some structural issues in the existing tests that needs to be fixed or we have bigger compatibility issues in terms of package not compatible with the new kafka version, requiring further code changes?

I will work to priorities this accordingly.

[khushijain21]

Hello @lalit-satapathy, I think it is a structural issue as I have seen similar behaviour for version 2.9 as well. You may prioritize this accordingly, thanks!

[stefans-elastic]

I think I've figured the solution for this change (MODULE=kafka mage goIntegTest tests pass locally). For Illustration I've opened a PR with comments describing the additional changes that I've made - #42286.
One thing that I didn't mention in my PR - I had to uncomment that bock of code in libbeat/tests/compose/compose.go (it didn't work otherwise)

[lalit-satapathy]

This has bee closed as @stefans-elastic is raising a new PR #42286

[stefans-elastic]

Closing this PR in favor of #42286