feat(kubernetes) Support Tolerations in Pod spec

[jgramoll]

Allow adding:

kubernetes:
  tolerations:
    - effect: NoSchedule
      key: foobar
      operator: Equal
      value: "true"

[spinnakerbot]

The following commits need their title changed:

• fc3bb37: Merge pull request #3 from spinnaker/master

• aa0413e: feat(kubernetes)

Please format your commit title into the form:

<type>(<scope>): <subject>, e.g. fix(kubernetes): address NPE in status check

This allows us to easily generate changelogs & determine semantic version numbers when cutting releases. You can read more about commit conventions here.

[ezimanyi]

Thanks for the PR! Isn't it already possible to specify tolerations by setting them under deploymentEnvironment after the changes in #1396? Or is the issue that you'd like a way to specify tolerations that apply to all services (rather than have to specify one-by-one which I think you might need to do)?

[jgramoll]

So #1396 allowed for specifying tolerations on deploymentEnvironment which applies the toleration to all pods in the deployment.

We only have a taint on one pod (clouddriver) and need to be able to specify the toleration for only that pod. This provides that by updating getTolerations to check for the pod spec before checking the deploymentEnvironment spec

[ezimanyi]

@jgramoll : I think I'm still confused about when you'd want to set this on deploymentEnvironment vs. under kubernetes. It seems like #1396 allows you to set:

deploymentEnvironment:
  tolerations:
    clouddriver: {} // set tolerations here

to specifically set tolerations for clouddriver. If I'm understanding your above comment correctly, that's what you're trying to do---apply a toleration only to the clouddriver pods, but I may be mis-understanding.

The reason I'm asking is that it feels a bit confusing to have two different places to set the tolerations and I want to make sure we're clear about when users should set one vs the other (and document that). Thanks!

[jgramoll]

I didn't know that deploymentEnvironment could allow tolerations to be set for single service. I thought it had to be

deploymentEnvironment:
  tolerations:
    - effect: NoSchedule
    ...

So my next question is, we currently have

kubernetes:
  nodeSelector:
    kops.k8s.io/instancegroup: clouddriver

but I couldn't get

deploymentEnvironment:
    nodeSelectors:
      clouddriver:
        kops.k8s.io/instancegroup: clouddriver

to work. I would like to have the settings live side by side. Is deploymentEnvironment then the preferred location, and I should try and get node selectors to work for a single service?

[ezimanyi]

I'll caveat that I haven't personally tried to set tolerations myself in deploymentEnvironment but I have set other properties there on a per-microservice basis and the code for tolerations looks to be doing the same thing. So I'm pretty sure that setting tolerations there should work, but definitely let me know if it doesn't.

It looks like nodeSelectors is a field in deploymentEnvironment though it doesn't support being set on a per-microservice basis. Interestingly though, V2 deployments completely ignore the field and just grab the one from kubernetes. We should probably also look in deploymentEnvironment here.

So maybe the answer is to make the deploymentEvironment nodeSelectors work and then you'd be able to define both there? That being said I don't really have a strong opinion about which place is better, and am not sure I fully understand the history, so am fine with either.

[Eslamanwar]

• I suggest to add the toleration in this list
  https://www.spinnaker.io/reference/halyard/custom/#kubernetes
  or add hal command to add toleration to services
  so we can use in the helm chart (https://github.com/helm/charts/tree/master/stable/spinnaker)and add
  toleration in the additionalServiceSettings: part like this
  additionalServiceSettings:
  clouddriver.yml:
  kubernetes:
  toleration:
  - key: dubizzle.com/group
  operator: Equal
  value: application

for now i just can do this :
additionalServiceSettings:
clouddriver.yml:
kubernetes:
nodeSelector:
dubizzle.com/group: application

[Badbond]

We have had similar issues. If you use (derivatives of) the Helm chart @Eslamanwar mentions, there are only two ways to control the K8s settings of Spinnaker services:

• Through hal CLI commands, which only supports a subset (e.g. component-sizing).
• Through service-settings YAML files, which only supports a subset of the fields.

Unfortunately in both paths it is not possible to set up tolerations. The same accounts for node affinity. Note, tweaking the deploymentEnvironment of the Halyard config is not possible as this is constructed through the hal CLI commands of the chart.

We now have a hacky workaround using Kyverno mutation policies to modify the K8s resources when the resources are scheduled for modification.