Build Strimzi with a custom Kafka version

[ThePoipoi]

Hi everyone. Quick question today, to understand if it's possible.

I have retrieved the Apache Kafka project on my own, and build it.
At the end, it produce a .tgz file that I want to use in my Strimzi build.
To do that, I have modify the kafka-versions.yaml file as follow (after removed each other Kafka version listed):

- version: 3.0.0
  format: 3.0
  protocol: 3.0
  url: file://C:/Users/me/kafka/core/build/distributions/kafka_2.13-3.0.0-SNAPSHOT.tgz
  zookeeper: 3.5.9
  third-party-libs: 3.0.x
  supported: true
  default: true

But during the build, I've got the following error message:

Could not resolve dependencies for project io.strimzi:config-model-generator:jar:0.22.0-SNAPSHOT: 
The following artifacts could not be resolved: org.apache.kafka:kafka_2.12:jar:3.0.0, org.apache.kafka:kafka-clients:jar:3.0.0: 
Failure to find org.apache.kafka:kafka_2.12:jar:3.0.0 in https://[MY_SERVEUR_URL]:8443/repository/public/ was cached
in the local repository, resolution will not be reattempted until the update interval of my-mirror has elapsed or updates are forced 

Can you tell me why it still trying to use an other version that the one I gave ?
Or if I'm just doing some crap ?

Thanks a lot.

[scholzj]

I do not think you can add Kafka versions which don't exist so easily. The part which is failing is trying to get the configuration options for the version you added. What you can try is to push the JARs into some Maven repository and add it as custom repository to the pom.xml. You will obviously also add the working 3rd party libraries. And TBH I'm not sure curl can deal with the file:// URL ... I never tried it, so not sure it works.

[tomncooper]

The file:// for the Kafka dist used in the image should work fine. But Jakub is right that you will need to publish the Kafka 3.0.0 artifacts to your local maven repository (there should be a gradle task for that in the Kafka build) and create the corresponding 3.0.x pom file in docker-images/kafka/kafka-third-party-libs that lists all the correct dependency versions.

[ThePoipoi]

Ok thanks guys. I'll try and let you know

[ThePoipoi]

Hey guys
Some news. It's seems to work, not necessarily in 3.0.0, but I've have build both Kafka and Strimzi with my "own" code in 2.7.0_2.13
I still have some issues with the operator docker image built, that no work as expected, but I'll see that later.

For the moment, I have another issue, most "stupid": I can't successfully built the last 0.21.1 Strimzi version !
It's really weird, because I have just retrieved this version, then use

• docker login
• export both DOCKER_ORG and DOCKER_REGISTRY
• run "ake MVN_ARGS='-DskipTests -DskipIT' all"

without changing anything.
The build run for like 30/40min, create 4 docker images

• strimzi/jmxtrans (latest)
• strimzi/operator (latest)
• my_repo/my_login/operator (latest)
• strimzi/base (latest)

then always failed with:

# Tag the latest image we built with the given latest tag
docker tag strimzi/operator:latest my_repo/my_login/operator:latest
# Push the latest-tagged image to the registry
docker push my_repo/my_login/operator:latest
The push refers to repository [docker.io/my_repo/my_login/operator]
6229cb0bc4e2: Preparing 
4c05b8386258: Preparing 
c14e40a00603: Preparing 
d55e7485ff90: Preparing 
f8e0fabb4d52: Preparing 
80be7a4bffb1: Waiting 
6deb95bcad13: Waiting 
174f56854903: Waiting 
denied: requested access to the resource is denied
make[2]: *** [docker_push_default] Erreur 1
make[2] : on quitte le répertoire « /user_me/Documents/workspace/strimzi-kafka-operator-0.21.1-test/docker-images/operator »
make[1]: *** [docker_push] Erreur 2
make[1] : on quitte le répertoire « /user_me/Documents/workspace/strimzi-kafka-operator-0.21.1-test/docker-images »
make: *** [docker-images] Erreur 2

Any idea why ?

[ThePoipoi]

Ok, I'm back faster than expected. Even with the 0.22.0 version it's failing:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:testCompile (default-testCompile) on project api-conversion: Compilation failure
[ERROR] /user_me/Documents/workspace/strimzi-kafka-operator-0.22.0-test/api-conversion/src/test/java/io/strimzi/kafka/api/conversion/cli/CrdUpgradeCommandIT.java:[77,37] cannot find symbol
[ERROR]   symbol:   method withNewConfigMapKeyRef(java.lang.String,java.lang.String,boolean)
[ERROR]   location: interface io.strimzi.api.kafka.model.JmxPrometheusExporterMetricsFluent.ValueFromNested<io.strimzi.api.kafka.model.ZookeeperClusterSpecFluent.JmxPrometheusExporterMetricsConfigNested<io.strimzi.api.kafka.model.KafkaSpecFluent.ZookeeperNested<io.strimzi.api.kafka.model.KafkaFluent.SpecNested<io.strimzi.api.kafka.model.KafkaBuilder>>>>

And I'm running:

make MVN_ARGS='-DskipTests -DskipIT' all

So it's shouldn't run test at all and thrown this error, right ?
So where does this error come from ?

[scholzj]

This seems to work fine for me locally as well as on the CI. So not sure why you seem to be missing the generated classes. Have you done mvn clean after rebasing to 0.22? There is a lot of changes.

[ThePoipoi]

Yes. And it's worst than that, because I have :

• mvn clean
• remove my local ".m2/repository/io/strimzi" folder
• mvn install -DskipTests -DskipITs

And still have the issue...

This one is a tough one

[scholzj]

Maybe some OpenJDK differences? I have

java -version
openjdk version "11.0.9.1" 2020-11-04
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.9.1+1)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.9.1+1, mixed mode)

[scholzj]

I opened #4596 which might fix it. But it is hard to test for me since the error does not happen for me. Any chance you can try if the branch from #4596 builds correctly in this environment where you had this error before?

[ThePoipoi]

Ok so, I've update everything on my VM, re-download the 0.22.0 zip, and retry from scratch.
It's now working well !!!

Everything is fine, and the last (but small) issue is already known (role/rolebinding on this specific version).
Thanks a lot Jakub for your help, it's just perfect ;)

[scholzj]

It is still weird why the compilation did not worked. But at least it works now.

[ThePoipoi]

Yes, I don't understand neither. But as you said, it works :)

Thanks again !