Mixed worker type cluster

[CGe0516]

Hello! I'm working to extend the SpecCluster class to support multiple worker types and was hoping to see if you all had any thoughts on how best to do this. Currently, what I'm doing is I created a concept of 'worker types'. I consider the worker_spec fed in through the worker param to be the spec for a 'default' worker. I then have the cluster store a dict of alternate_worker_args (worker_type -> worker_spec). I also modify the scale() and adapt() functionality and add a scale_worker() method.

Here's what I was thinking so far:

• Cluster(worker=worker_kwargs, alternate_workers = {‘worker_name’: worker_kwargs, ...}), will refer to worker as default_worker
• Cluster.scale(n): n > total current workers: add default_workers until there are n total, n < total current workers: start by removing default workers, then remove alternate_workers until there are n total
• Cluster.scale_worker(worker_name, n): add or remove workers until there are n of type worker_name, can also specify worker =’default’ to add/remove default workers
• Cluster.adapt(worker_types_to_adapt=['default']): after determining a target number, consider workers with types in worker_types_to_adapt for removal.

Do you think this would work well for users or do you think this might create some new pain points? Any ideas for how this could be better designed?

[SultanOrazbayev]

To avoid modifying existing Cluster functions, maybe it's worth adding new functions:

• Cluster.add_alternate_workers(worker_kwargs)
• Cluster.scale_alternate_workers(worker_name, n)
• Cluster.adapt_alternate_workers(worker_name)

(or some other naming convention, like Cluster.alt_workers_add/scale/adapt())

But this is only as a suggestion, I'm not sure what's better. In the past whenever I had workflows where I could use multiple types of workers, I ended up splitting the code into separate parts... maybe these functions will only be needed in very special cases, and then there is some benefit in keeping them as separate functions.