update kafka version used in testing to 3.6.0

[stefans-elastic]

Proposed commit message

Updated kafka version used in our integration tests

Based on #41833

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Kafka version 4.0 will not be compatible with client libraries using protocol version less than 2.1.0

Related issues

Closes #41654

[botelastic]

This pull request doesn't have a Team:<team> label.

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @stefans-elastic? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[khushijain21]

could you also update the kafka version here, https://github.com/elastic/beats/blob/main/metricbeat/module/kafka/partition/partition.go#L54

[stefans-elastic]

could you also update the kafka version here, https://github.com/elastic/beats/blob/main/metricbeat/module/kafka/partition/partition.go#L54

sure
updated the version

[stefans-elastic]

@elastic/elastic-agent-data-plane and @elastic/obs-infraobs-integrations could you review this PR?

[khushijain21]

You may also want to update the docs. Please refer to this PR #17819.

[stefans-elastic]

You may also want to update the docs. Please refer to this PR #17819.

done

[Linu-Elias]

LGTM