feat: add support for vendor-specific topic configuration

[gokerakc]

About the PR

Currently, only 5 topic configuration properties are supported by the asyncapi kafka binding (json schema here).

These properties are;

• cleanup.policy
• retention.ms
• retention.bytes
• delete.retention.ms
• max.message.bytes

But in some cases, there is a need to specify vendor-specific topic configurations (e.g. confluent-topic-configs). Currently, there is no way to set custom configs for the topicConfigurationObject.

I introduced custom.configs property to solve that issue. This is an additional property for the topicConfigurationObject and additional properties will be enabled in the asyncapi binding repo as well (please see the related PR) so there are no impediments for us to use that additional property.

I hope the description was clear and please let me know if you have any questions/suggestions.

[dpwdec]

LGTM! Very much needed given the lack of support in the binding.

[VisualBean]

I see this is a breaking change (property names changed), generally fine, but be sure to mention it in the description - and title so we get it documented.
Anyway, isee that the binding pr won't get approved until after V3.
So I'm the official capacity, we can't really add this in.

However - I would have assumed that the vendor specifics would be solved through binding extensions (which should be supported). Is there a reason for adding these outside of that?

I'm not against the change btw - just curious - I think we can add it in, as a nice to have.

I will wait for the next major Asyncapi.net version due to the breaking nature of the pr. - should be within the week if all goes well 👍

[gokerakc]

Hi @VisualBean,

Thanks for the quick response. Currently, extensions are not enabled for the topicConfigurationObject and I agree that extensions will solve the same problem.

Initially, I came up with the custom.configs property to make the intention clear and let other developers know that they can use this property to parse the unsupported configs. But on second thought, I think I like the extension approach better.

If you are okay with it I can update this PR and enable the extensions for the TopicConfigurationObject?

[VisualBean]

I think it fits nicely with the intended use of extensions. Although it doesn't state anywhere that they are enabled for sub properties, I don't see how that couldn't be the case.

[gokerakc]

I've closed this PR since the asyncapi kafka bindings need to be updated first. I'll revisit this branch and create a new PR once asyncapi kafka bindings have support to handle the custom topic config cases.

[VisualBean]

I am fine with adding the extensions up front. - I wont be able to make support for 3.0 any time soon anyway.
I do have some other stuff in the pipeline though, for a new major version of AsyncApi.Net (mostly minor stuff, but breaking).
So if you need it in, make a PR towards the vnext branch

[gokerakc]

That's nice, thanks for the information. I have another question and a suggestion about the solution.

It's most likely that additionalProperties will be enabled for the topicConfigurationObject in the asyncapi bindings (this line). In my conversations with the asyncapi team, they agreed on enabling that to give flexibility to add extra topic configurations.

Do we have current support for the additionalProperties in AsyncAPI.NET? Instead of adding extensions, I think enabling additionalProperties for that object would be a better solution. What are your thoughts about it?

[VisualBean]

additionalProperties as in the JsonSchema additionalProperties?

I'm not a big fan of JsonSchema, mostly because it's a PITA to implement and it has some interoperability issues.

It shouldn't be a problem supporting it here though, the types are there for jsonSchema - so it should be easy enough

[gokerakc]

additionalProperties as in the JsonSchema additionalProperties?

Yes, it is the additionalProperties as in the jsonSchema. This is the line that will get updated in the asyncapi repo -> channel.json#31

If you are happy with it I can help you implement that so the tooling can handle the additional topic config properties.

[gokerakc]

What do you think about adding support for additionalProperties @VisualBean ? Or a functionality that lets you extend/override the binding objects so users can customise the existing bindings if they need to?

[VisualBean]

I think both could work.
Personally, i think extensions is more on point for this sort of thing.
but additionalProperties also has a nice ring to it. ill support both, so which ever feels better using)

[gokerakc]

That sounds great!

I did check the codebase to implement additionalProperties functionality but couldn't come up with a solution yet. Do you have anything in your mind to give it a start?