Assess upgrade readiness before running an upgrade

[marians]

User Story

• As a cluster admin, I want to verify easily that a cluster meets all requirements to endure an upgrade.

Details, Background

In our workload cluster upgrade docs we collect a number of best practices and recommendations to help ensure that workload clusters and workloads are prepared for a cluster upgrade.

There is derailed/popeye as a CLI tool to vet clusters against a number of rules. It is licensed under the Apache 2.0 licence. Users can configure it to mute or re-configure certain checks.

We have already contributed to popeye in order to perform the checks for what we consider vital before a workload cluster upgrade, and will continue to do so.

Checks to add

• Consider high-availability masters (a. k. a. control plane nodes) Check for high availability masters derailed/popeye#136 Add check for the number of master nodes derailed/popeye#131
• Avoid pods without owner reference (e. g. ReplicaSet, DaemonSet, StatefulSet) Add check for uncontrolled pods derailed/popeye#137 Add check for pod being controlled by other resource derailed/popeye#132
• Ensure redundancy for workloads
  • Services: Add check for Service with a single endpoint derailed/popeye#138, Add check for service having multiple endpoints derailed/popeye#135
• Distribute workloads across nodes Check whether replicas are all on the same node derailed/popeye#143 Replicas on same node derailed/popeye#149
• Handle termination signals in Pods
• Avoid ephemeral resources
  • emptyDir volumes
  • CSI ephemeral volumes
  • generic ephemeral volumes
• Configure webhook timeouts
• Check that the images needed are available in the registry
• Make Docker images as small as possible

Long term perspective

If the checks provide useful when performed manually by Giant Swarm staff and customers before performing an upgrade, we can consider running popeye in the cluster and providing results in a standard way.

This can either happen as a permanent service, potentially as an app to be installed from a Giant Swarm catalog, or as a necessary step before an upgrade (executed via a core component / default app).

[cokiengchiara]

sig product Mon, 03. May 2021

• needs a team.
• @alex-dabija will take to @giantswarm/team-celestial , because they are already taking care of upgrades.