Create a MirrorMaker configuration to append a prefix to the topic names in the destination

[skas-zilo]

Hi,

I'm using Strimzi to manage a Kafka cluster. I'm trying to create a Kafka MirrorMaker configuration to sync topic "a" to topic "b" in another cluster. I'm using the configuration below to append a prefix in the destination. I've found two things that seem odd, and I'm not sure if this is the correct behavior from Kafka or not.

Topics are not created automatically in the destination. I get the error "unknown_topic_or_partition." I have to create the topic manually in the destination to solve the issue. I think it should be created automatically. Please suggest a solution.

The original topic name is shown in the destination cluster instead of the one that I need. For example, I need "a" to become "1b," but I get "a" to "a" instead.

        transforms: "renameDev"
        transforms.renameDev.type: "org.apache.kafka.connect.transforms.RegexRouter"
        transforms.renameDev.regex: "^(dev-1)$"
        transforms.renameDev.replacement: "test-$1"

[im-konge]

Topics are not created automatically in the destination. I get the error "unknown_topic_or_partition." I have to create the topic manually in the destination to solve the issue. I think it should be created automatically. Please suggest a solution.

I'm not expert in MM1, but I think that you can use auto.create.topics.enable for automatic creation of the topics with MM1? I read somewhere that the topics should be created first with MM1 in Kafka, but maybe I'm wrong.

The original topic name is shown in the destination cluster instead of the one that I need. For example, I need "a" to become "1b," but I get "a" to "a" instead.

Again, I didn't try that, but I guess that in case that you do test-$1 it will be only prefixed with test-, but only in case that the regex you specified there will match something (FMPOV it should match only dev-1, so the result should be test-dev-1) - that way I mean that you will not do a to 1b, but for example a to 1a (if the 1 will be the prefix you have there). It's hard to say if there is anything else that could be wrong. Maybe the regex is not matching anything? Maybe there is other misconfiguration of MM?

[skas-zilo]

@im-konge Actually, I'm using MM2. Is there a way for MM2 to append a prefix to the destination topics?

[im-konge]

You can specify replication policy, default for MM2 is org.apache.kafka.connect.mirror.DefaultReplicationPolicy. There you can configure the separator. But I guess that if you need some special handling and the "separator" will not be exactly what you want, you can write your own policy and use it in MM2.

[im-konge]

In the past, when we had our own implementation of IdentityReplicationPolicy - https://github.com/strimzi/mirror-maker-2-extensions (it's archived now as we are using the official Kafka one).
But you can get an idea on how it can be done (maybe from the previous PRs, the latest code just extends the official IdentityReplicationPolicy class).

[skas-zilo]

Thank you for your guidance. I was able to get it to work by using DefaultReplicationPolicy. It appends the prefix from the source name to the destination topic names.

[im-konge]

Perfect! :)