Not able to achieve successful binding rate beyond ~300 pods/sec

[rishabh325]

Not able to achieve successful binding rate beyond ~300 pods/sec. When running binder in active-active mode, getting high conflict rate while in active-passive mode overall binding rate is below ~300 pods/sec

Configurations:
Nodes: 30k
Pods: ~150k
Creation Rate: ~1.75k pods/sec via clusterloader

Case 1:

Service	Leader Elected	Instances	Resource Limit
Dispatcher	No	4	2 instances w/ 32 cores/180Gi
Scheduler	No	2	2 instances w/ 32 cores/250Gi
Binder	No	4	2 instances w/ 32 cores/180Gi

.

Case 2:

Service	Leader Elected	Instances	Resource Limit
Dispatcher	No	4	2 instances w/ 32 cores/180Gi
Scheduler	No	2	2 instances w/ 32 cores/250Gi
Binder	Yes	2	2 instances w/ 32 cores/180Gi

[binacs]

Apologies for the temporary absence of a deployment guide for multi-instance setups, which may have caused confusion. We are working to improve this documentation as soon as possible.

In the architecture of the Godel distributed scheduler, only one dispatcher and one binder instance are expected to be active at any given time. Multiple dispatcher/binder instances are deployed for high availability and must utilize leader election to prevent conflicts.

For multiple scheduler instances, there are two possible scenarios:

1. If multiple instances belong to the same shard, they should share the same scheduler name and enable leader election.
2. If instances belong to different shards, each shard's instances should be assigned a unique scheduler name.

If you have any further questions, please feel free to continue the discussion in the issue.