How worker to worker nodes communication works in Kamaji?

[sahidrahman404]

Hello. I'm still new to Kubernetes, and I don't understand how worker nodes communicate with other worker nodes in Kamji. Do they communicate through the Konnectivity agent on each worker node without going through the control planes first? Also, I want to ask if it's okay to have a central Tenant Control Plane in one region, for example, Vultr Singapore, and have worker nodes in another region, such as Hetzner Finland?

[prometherion]

The communication between the Worker nodes and the Control Plane is definitely the same as you would have on a traditional Kubernetes cluster: the Control Plane endpoint refers to the IP used by your Tenant Control Plane.

This is trivial in flat networks, things get more complicated if the Control Plane is deployed in a private subnet, different from the worker nodes ones: and there are many reasons for this kind of setup, mostly for security reasons. In a such case, the worker nodes wouldn't be routable from the Kubernetes API Server, and here comes the konnectivity integration which provides a reverse tunnel to create a bi-directional communication between the API Server (which would be exposed with a "public" IP) and the worker nodes that must have a connection to the API Server for the bootstrap process.

About the last question, there are no big issues with latency between the CP and WN although some considerations must be put in place:

• fine-tune the API Server --kubelet-timeout flag, default to 5 seconds: otherwise, the operations would timeout and mark the node as Unknown or Unreachable
• the potential communication lag between CP and WN operations, such as exec, logs, etc.: although a tunnel is established by konnectivity, you have to consider the round trip time required.

The design you suggested is ok from a theoretical point of view, although this implies some variables out of your control, such as the WAN, such as network failures, or the physical layer issues (sharks love to crunch cables). As I always suggest, it's possible, you just need to evaluate the pros and cons.

[sahidrahman404]

Wow, thanks for your thorough explanation! After watching videos about Konnectivity, it seems that Google also uses this feature for their Control Planes. I found this project from your Reddit post. I don't understand why people have bad takes on such a cool project like this. Just realized that sharks dine on fiber optic cables. 😂

[prometherion]

I don't understand why people have bad takes

It's reddit, mate.