Build Docker image with class collision

[guwei0kanyun]

Info

Strimzi Version: 0.45.x

Question

When we build operator docker image with command:

DOCKER_ORG=xx DOCKER_PLATFORM="xx" DOCKER_REGISTRY=xx DOCKER_TAG=xx make MVN_ARGS='-DskipTests' all

It gives errors below
../find-classpath-collision.sh strimzi/kafka:build-kafka-3.8.0 /opt/kafka/libs "tmp/3.8.0.ignorelist"
Successfully copied 152MB to /var/folders/yw/yt4c5wj12hdc5kvyz0d4lkzm0000gn/T/tmp.rbzuHCX5ib
ERROR: Different class files with same name from different jars found!
kafka-server-common-3.8.0.jar org/apache/kafka/timeline/SnapshotRegistry.class
kafka-server-common-3.9.0.jar org/apache/kafka/timeline/SnapshotRegistry.class

It seems there are class collisions.

How can we find out which lib jar cause this problem?

Any Tips are appreciated
Thanks

[im-konge]

Hi, did you try to run make clean before the make all?
This can happen when you have there old binaries for the previous versions.
I had similar issue when adding new version of plugin, I had to clean it, especially the binaries folder.

[guwei0kanyun]

I tried it and end with the same error. It seems this is not the solution for my case.

But anyway thanks for your suggestion.

[im-konge]

Hmm it should clear everything really.
Did you added some plugin that would download it again?

[guwei0kanyun]

yes, we plugin cruise-control in our cluster.
For Broker depends on cruise-control metric reporter, so we update cruise-control's version which is compatible with 3.9.0 kafka.

I have tried exclude kafka-server-common below cruise-control dependency, and it didn't work.

[guwei0kanyun]

As a temporary and somewhat risky solution, adding

kafka-server-common-3.9.0.jar
kafka-storage-3.9.0.jar
kafka-raft-3.9.0.jar
kafka-server-3.9.0.jar
kafka-storage-api-3.9.0.jar
kafka-metadata-3.9.0.jar
kafka-transaction-coordinator-3.9.0.jar
kafka-group-coordinator-3.9.0.jar
kafka-group-coordinator-api-3.9.0.jar
kafka-server-common-3.8.0.jar
kafka-storage-3.8.0.jar
kafka-raft-3.8.0.jar
kafka-server-3.8.0.jar
kafka-storage-api-3.8.0.jar
kafka-metadata-3.8.0.jar
kafka-transaction-coordinator-3.8.0.jar
kafka-group-coordinator-3.8.0.jar
kafka-group-coordinator-api-3.8.0.jar
kafka-server-common-3.8.1.jar
kafka-storage-3.8.1.jar
kafka-raft-3.8.1.jar
kafka-server-3.8.1.jar
kafka-storage-api-3.8.1.jar
kafka-metadata-3.8.1.jar
kafka-transaction-coordinator-3.8.1.jar
kafka-group-coordinator-3.8.1.jar
kafka-group-coordinator-api-3.8.1.jar

in docker-images/kafka-based/kafka/tmp/3.8.0.ignorelist and docker-images/kafka-based/kafka/tmp/3.8.1.ignorelist can bypass this error and prevent the blockage of the image build.

docker-images/kafka-based/kafka/tmp/3.9.0.ignorelist remains as it was is ok. There is no class collisions in building 3.9.0 version image

[ShubhamRwt]

Do you want to use your own cruise-control-metrics-reporter. jar or are you just trying to change the version of metrics-reporter in the third party libs? @guwei0kanyun.

[guwei0kanyun]

It's inside docker-images/artifacts/kafka-thirdparty-libs/3.9.x/pom.xml It also depends on which version of Strimzi you are trying to do this. I guess that we should have there correct version of CC for particular Kafka version.

@ppatierno @kyguy @ShubhamRwt WDYT?

EDIT: I see that you are on 0.45.0 🤦

Yes, we change the cc version to the one compatible with 3.9.0 kafka in follow files

• docker-images/artifacts/kafka-thirdparty-libs/3.9.x/pom.xml
• docker-images/artifacts/kafka-thirdparty-libs/3.8.x/pom.xml
• docker-images/artifacts/kafka-thirdparty-libs/cc/pom.xml

[guwei0kanyun]

Do you want to use your own cruise-control-metrics-reporter. jar or are you just trying to change the version of metrics-reporter in the third party libs? @guwei0kanyun.

We integrate the whole cc in strimzi. The reporter is part of it I think?
Particularly we maintain our branch followed from https://github.com/linkedin/cruise-control, and using it in kafka-thirdparty-libs/xx/pom.xml

[ShubhamRwt]

Looking into your ignore list file, why do you have to add a lot of these Kafka related jars? Are you sure you are just overriding the version of cruise-control-metrics-reporter to your desired version in the 3.8.0 pom file since we only include the metrics reporter jar in the kafka libs as it runs on broker side

[guwei0kanyun]

Looking into your ignore list file, why do you have to add a lot of these Kafka related jars? Are you sure you are just overriding the version of cruise-control-metrics-reporter to your desired version in the 3.8.0 pom file since we only include the metrics reporter jar in the kafka libs as it runs on broker side

1. Because the error log clearly mentions that some classes in these jars are "Different class files with same name from different jars found!"
2. Yes, I only changed the version below in docker-images/artifacts/kafka-thirdparty-libs/3.8.x/pom.xml by checking git history of the file

<dependency>
      <groupId>com.rockee.cruisecontrol</groupId>
      <artifactId>cruise-control-metrics-reporter</artifactId>
      <version>${cruise-control.version}</version>
      <exclusions>
          <exclusion>
              <groupId>org.apache.kafka</groupId>
              <artifactId>kafka_2.13</artifactId>
          </exclusion>
          <exclusion>
              <groupId>org.apache.kafka</groupId>
              <artifactId>kafka-clients</artifactId>
          </exclusion>
          <exclusion>
              <groupId>org.slf4j</groupId>
              <artifactId>slf4j-api</artifactId>
          </exclusion>
          <exclusion>
              <groupId>org.apache.logging.log4j</groupId>
              <artifactId>log4j-slf4j-impl</artifactId>
          </exclusion>
          <exclusion>
              <groupId>com.fasterxml.jackson.core</groupId>
              <artifactId>jackson-databind</artifactId>
          </exclusion>
      </exclusions>
  </dependency>