Custom infrastructure provider support

[andreykont]

Hello.
What about if KCP will support for any type of infrastructure provider?

Any infrastructure provider has to support this specification to it's infrastructure cluster.
It seems that in spite of infrastructure type KCP does almost the same. KCP patches controlPlaneEndpoint host and port.

This feature will make it possible to use custom infrastructure providers together with KCP provider.

[prometherion]

Thanks for starting the discussion, @andreykont: the issue has been migrated here since we'd like to keep the Issue section just for bug reports.

I couldn't agree more with you, and I'll keep my experience short on providing support for the Kamaji Control Plane provider to multiple Infrastructure providers.

Even tho we have some Cluster API contracts, it's still not yet clear to me how to deal with the externally managed Control Planes, such as Kamaji. The first implementation has been achieved with the CAPO community and we agreed to have the infrastructure Cluster patched by Kamaji since it's the duty to create the Load Balancer: even tho all the current implementations are doing so, I'm not really sure this is something broadly accepted by the CAPI community, also considering that in most of the infrastructure Providers the /spec/controlPlaneEndpoint is requiring a value before the creation, something that wouldn't fit with the Kamaji behaviour which is creating the LB after the TenantControlPlane has been applied, which is done by the kamaji-control-plane-provider once the Infrastructure cluster has been marked ready.

With that said, my personal idea was to clearly state what we're supporting, and what we don't, just to prevent potential issues with adopters complaining it's not working as expected.

Trying to summarize an outcome from this, I would propose trying to engage with the Cluster API folks and formalize what we would expect from the whole community to have a fully and comformant support not only for Kamaji itself, but for all the externally managed control plane providers. This would require having several meeting with the community, explaining what we've done, and what we're pointing to, as well as educating the whole community projects in getting ready in supporting this.

Let me know if you're up in helping us on this.

[andreykont]

Yes, I inspired in Kamaji project approache. And I want to help. I have some idea. And ready to make a proposal.

[andreykont]

As I said before I have an idea. And I'm ready to write a proposal with user stories and some explanation. But let's discuss the general idea of that proposal.
The main problem is that Kamaji provides kube-api load-balancer and CAPI Infra cluster does the same.

I think kamaji-control-plane-provider should not patch Infrastructure cluster because:

1. patching objects owned by the other controller is not a good idea. Any operator has to modify only it's own CRs. (maybe some metadata, OwnerReference for example)
2. CR's specification (.spec) should have object's desired state on creation.

User story 1:
I want to use standard Infrastructure provider (cluster-api-provider-vsphere for example).
I set kube-api endpoint address to infra cluster spec on creation.
kamaji-control-plane-provider propagates endpoint address from infra cluster to the TCP provider endpoint parameter.
Even though, I have to set desired external IP address with some service annotations.
Kamaji should not wait for getting external IP address for the load balancer service.
Kamaji uses TCP provider endpoint parameter instead.

Required Kamaji functionality:

1. TCP supports provider endpoint parameter.

Required kamaji-control-plane-provider functionality:

1. Propagate infrastructure cluster endpoint (hostname or IP) into TCP provider endpoint parameter.

[prometherion]

1. patching objects owned by the other controller is not a good idea. Any operator has to modify only it's own CRs. (maybe some metadata, OwnerReference for example)
2. CR's specification (.spec) should have object's desired state on creation

That's not entirely correct, I'm recalling some providers such as the vSphere one that accepts an empty VSphereCluster.spec.controlPlaneEndpoint field waiting to get a VIP, from the user, or rather, from the NSX.

Pretty similar with CAPO, although by using a different approach, such as disabling the floating IP (OpenStackCluster.spec.disableAPIServerFloatingIP) and disabling the API Server Load Balancer reconciliation (OpenStackCluster.spec.apiServerLoadBalancer.enabled).

If we take for granted that everything must be known at creation time, we lack the chance of having a dynamic IP assigned at creation time.

---

However, I'm really open to getting this issue solved at the Cluster API level community, rather than by assumptions. AFAIK, Cluster API is already aware if the control plane is externally managed (e.g. not managed as a machine) thanks to the ControlPlane status subresource field externalManagedControlPlane.

However, it seems only related to machines, wondering if we should start a conversation about it since there's an overlap with LBs too: having a Control Plane means I have an endpoint to consume it, so there's an overlap with the infrastructure provider duties.

[andreykont]

Hello. Ok... I guess you agree, my approach doesn't support assign dynamic IP address at creation time.
But, What the next action? To make this topic till the end.

[andreykont]

Hello. Is this topic still actual?
Due to this kamaji-cp doesn't depend on Infra Provider any more.

[prometherion]

Thanks for the heads up, Andrey! Definitely an interesting advancement in regard to Kamaji.

I'll try to dig this further, even tho I'm a bit worried we would need newer versions of the Infrastructure providers supporting this feature, but I could be wrong.

[prometherion]

Implemented with #137.