Team based Kubernetes Node Selectors

[tlawrie]

Is your request related to a problem? Please describe.
Currently the only way to define where a task runs is the node selector at install time for all tasks across the instance

Describe the solution you'd like
Allow individual teams to have their own node selectors

Describe the benefits or justification for this request
Allows greater segregation and control of running tasks

Specific workflows could be GPU or client specific at the infra layer. Allowing a broader use case of the as a service and user needs for different types of workflows.

Additional context
Add any other context or screenshots about the feature request here.