Don't push unvalidated yamls to your cluster state repository (Post-mortem story)

[or-shachar]

Sharing with you a story:

TL;DR:: Pushing malformed yamls to cluster states can lead to serious problem. We need a way to validate the correctness of the cluster state repository before it is pushed, and before Flux components in the cluster need it.

How it went down

1. At my company, we use templates to generate Helmreleases objects for all of our charts.
2. Our charts come from "infra" "backend" and "frontend" categories.
3. I was given the task to migrate to Flux V2, change the templates and deploy the new toolkits.
4. At first - everything worked nicely and FluxV2 took over and deployed all the charts correctly.
5. At some point - I changed something in the templates that caused some indentation problems in the expanded yaml.
6. Unfortunately, there was nothing to stop it from being pushed to the GitRepository that was being watched by Flux
7. This caused a chain of failures in my cluster:
   • The kustomize controller tried to apply the new HelmRelease yaml.
   • It complained that cannot validate the data (see this slack thread)
   • I found the problem and fixed it
   • The helmreleased was fixed, but it started to complain about not seeing the charts, even though they were updated as ready (see this slack thread)
   • I figured that there was something stuck with source-controller and killed the pod.
   • The new pod refused to start (failed on healthchecks)
   • I uninstalled flux and reinstalled flux (using the flux cli utility)
   • None of the Flux pods managed to start (healthchecks)
   • I managed to extract error log saying : "Failed to get API Group-Resources" - it seemed like the components couldn't talk with the kubernetes service.

How it was fixed eventually

1. I re-installed my infra charts manually
2. Flux pods went back to life.

Why did it solve it?

• Part of my infra charts include ServiceAccounts, certificates , and other core objects. When I pushed helmrelease with bad format - they somehow got uninstalled, and then none of the components could talk with the kubernetes service.

Action items

• I probably want to validate the HelmRelease Yaml Before they are being pushed.

[stefanprodan]

You can validate locally and in CI all Flux custom resources using kubeval and our OpenAPI schemas. Example script here: https://github.com/fluxcd/flux2-kustomize-helm-example/blob/main/scripts/validate.sh

[or-shachar]

That's amazing! but I wonder if this could be integrated into the flux tool itself.

[stefanprodan]

I don’t know if it makes sense to create a kubeval clone in flux CLI. What issues you have with kubeval that flux would solve?

[or-shachar]

it's just yet another tool to install on my agent. I think that the tool requires for schemas to be in subdirectory called master-default-strict, otherwise - it skips schema validation.

I managed to write a small code in my HelmReleaseGen:

package validator

import (
	"fmt"
	"io/ioutil"
	"path"

	"github.com/instrumenta/kubeval/kubeval"
)

type FluxValidator struct {
	absolutePathToSchemas string
}

func (v FluxValidator) GetConfig(yamlPath string) kubeval.Config {
	return kubeval.Config{
		DefaultNamespace:          "default",
		KubernetesVersion:         "master",
		AdditionalSchemaLocations: []string{fmt.Sprintf("file://%s", v.absolutePathToSchemas)},
		Strict:                    true,
		FileName:                  yamlPath,
	}
}

func (v FluxValidator) ValidateFluxResource(yamlPath string) error {
	b, err := ioutil.ReadFile(yamlPath)
	if err != nil {
		panic(err)
	}
	config := v.GetConfig(yamlPath)
	_, err = kubeval.Validate(b, &config)
	return err
}

Such code can be part of Flux itself as flux-cli itself.