Cluster deployment using kubernetes

[dshah2012]

Hello @karimra ,

I am trying the cluster feature of the gnmic and have this doubt. I am building a helm chart that will deploy the gnmic to the kubernetes environmet in a cluster mode. I was looking at the examples of docker-compose.yaml for a reference. I see in the documentation that the at the start-up phase that the gnmic-api address are registered which would be different if we provide that for different gnmic instance. But the thing in deployment is that we specify the replicas and it will handle the replication for that pods/containers. In this case the GNMIC_API and GNMIC_CLUSTERING_INSTANCE_NAME and GNMIC_CLUSTERING_SERVICE_ADDRESS would remain same and just one instance would be registered. So does it mean that the leader will be just one as there are no other distinctive instances and are the targets been distributed in instance based upon the load irrespective of the GNMIC_CLUSTERING_INSTANCE_NAM, GNMIC_CLUSTERING_SERVICE_ADDRES ?

[karimra]

Hi @dshah2012 ,

The API port number does not have to be different, as long as the pods get different IPs.

The instance name and service address cannot be the same, there will be no clustering in this case.

I would run each gNMIc instance together with a consul instance in the same pod, in this case there is no need to specify an IP for gNMIc's API, a port number is enough, the Consul agent will add its own IP (which would also be gNMIc's) as the service IP.
In this case you also dont need to care about the service address.
Check Consul architecture if you haven't. You can run it in server mode or in client mode, you only need 1, 3 or 5 servers, all the others can be clients. Consul clients are light weight, so you can run them in all you pods.

As for the instance-name, you can use k8s initContainers to edit the config file before gNMIc container boots.

[karimra]

I still don't see any issue in the logs, can you share the relevant parts ? or the whole logs ?

[dshah2012]

Yes i don't see any error logs in the gnmic collector.
One of the things i see in consul server cluster is this.
Not sure is this related to the issue that I have.

agent.server.serf.lan: serf: EventMemberJoin: **hostname** 10.42.85.177
    2021-08-12T14:22:36.752Z [INFO]  agent.server: member joined, marking health alive: member=nso-pp-w-10.nso.int.bell.ca
    2021-08-12T14:22:36.754Z [ERROR] agent: gRPC conn opened but gRPC RPC is disabled, closing: conn=from=10.42.85.177:47464
    2021-08-12T14:22:37.757Z [ERROR] agent: gRPC conn opened but gRPC RPC is disabled, closing: conn=from=10.42.85.177:47488
    2021-08-12T14:22:38.142Z [WARN]  agent.server.kvs: Rejecting lock of key due to lock-delay: key=gnmic/cluster1/leader expire_time="2021-08-12 14:22:48.150539773 +0000 UTC m=+46004.080194988"
    2021-08-12T14:22:39.638Z [ERROR] agent: gRPC conn opened but gRPC RPC is disabled, closing: conn=from=10.42.85.177:47534
    2021-08-12T14:22:40.148Z [WARN]  agent.server.kvs: Rejecting lock of key due to lock-delay: key=gnmic/cluster1/leader expire_time="2021-08-12 14:22:48.150539773 +0000 UTC m=+46004.080194988"
    2021-08-12T14:22:41.830Z [ERROR] agent: gRPC conn opened but gRPC RPC is disabled, closing: conn=from=10.42.85.177:47578
    2021-08-12T14:22:42.153Z [WARN]  agent.server.kvs: Rejecting lock of key due to lock-delay: key=gnmic/cluster1/leader expire_time="2021-08-12 14:22:48.150539773 +0000 UTC m=+46004.080194988"
    2021-08-12T14:22:44.161Z [WARN]  agent.server.kvs: Rejecting lock of key due to lock-delay: key=gnmic/cluster1/leader expire_time="2021-08-12 14:22:48.150539773 +0000 UTC m=+46004.080194988"
    2021-08-12T14:22:46.166Z [WARN]  agent.server.kvs: Rejecting lock of key due to lock-delay: key=gnmic/cluster1/leader expire_time="2021-08-12 14:22:48.150539773 +0000 UTC m=+46004.080194988"
    2021-08-12T14:22:46.658Z [ERROR] agent: gRPC conn opened but gRPC RPC is disabled, closing: conn=from=10.42.85.177:47626
    2021-08-12T14:22:53.135Z [ERROR] agent: gRPC conn opened but gRPC RPC is disabled, closing: conn=from=10.42.85.177:47704
    2021-08-12T14:23:05.484Z [ERROR] agent: gRPC conn opened but gRPC RPC is disabled, closing: conn=from=10.42.85.177:47820
    2021-08-12T14:23:20.589Z [ERROR] agent: gRPC conn opened but gRPC RPC is disabled, closing: conn=from=10.42.85.177:47968
    2021-08-12T14:23:44.142Z [ERROR] agent: gRPC conn opened but gRPC RPC is disabled, closing: conn=from=10.42.85.177:48246
    2021-08-12T14:24:28.751Z [ERROR] agent: gRPC conn opened but gRPC RPC is disabled, closing: conn=from=10.42.85.177:48694

Still we are receiving the subscription response even after this error , so not sure whats the exact issue?

[karimra]

That means the leader key was not maintained by the initial leader, and another instance is trying to be come the leader.
it is being rejected because of the lock-delay, which is normal, it will just keep retrying.

Did you restart any of the gnmic instance?
It will be easier to understand what happened with all the logs, if you don't want to post them here, you can send them to me via email.

[dshah2012]

Sure. I can send them to you the fresh logs for both the consul server and gnmic instance.

[dshah2012]

I have send you the email with the logs for both.