Perform incremental upgrades for specific releases.

[gprossliner]

Hi guys!

On a test environment, I test how to upgrade a release of the harbor project chart (https://artifacthub.io/packages/helm/harbor/harbor).
Because it has not been upgraded a while, the installed version on production is 1.7.5.

If I use fluxcd for the upgrade, and I specify no version: 1.9* in the HelmRelease.spec.chart.spec.version field, the Helm-Controller successfully upgrades harbor to 1.9.0, as shown with helm history:

$ helm -n harbor history harbor                                
REVISION	UPDATED                 	STATUS    	CHART       	APP VERSION	DESCRIPTION     
1       	Thu May 12 14:51:38 2022	superseded	harbor-1.7.5	2.3.5      	Upgrade complete
2       	Thu May 12 14:55:26 2022	deployed  	harbor-1.9.0	2.5.0      	Upgrade complete

So far, so good.

But: Harbor has a known (and very annoying) bug, that if you skip a version, an error is logged in a component named harbor-core, telling it is unable to perform a database upgrade, and it creates a new (empty) database.

To avoid this, you need are not allowed to skip a version. If you take a look at the released versions of the chart (they are 1.7.5, 1.8.0, 1.8.1, 1.8.2, 1.9.0).

I would like to instruct flux to perform this when upgrading (e.g. HelmRelease.spec.upgrade.incremental, like):

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: harbor
  namespace: harbor
spec:
  interval: 1m0s
  upgrade:
    incremental: true #run in incremental mode -> no version is skipped
  chart:    
    spec:
      version: 1.9.*
      chart: harbor
      sourceRef:
        kind: HelmRepository
        name: helm-repo

The Helm-Controller should perform the upgrade to each version incrementally, resulting in the following history:

$ helm -n harbor history harbor                                
REVISION	UPDATED                 	STATUS    	CHART       	APP VERSION	DESCRIPTION     
1       	Thu May 12 15:21:38 2022	superseded	harbor-1.7.5	2.3.5      	Install complete
2       	Thu May 12 15:24:05 2022	superseded	harbor-1.8.0	2.4.0      	Upgrade complete
3       	Thu May 12 15:27:34 2022	superseded	harbor-1.8.1	2.4.1      	Upgrade complete
4       	Thu May 12 15:30:48 2022	superseded	harbor-1.8.2	2.4.2      	Upgrade complete
5       	Thu May 12 15:34:10 2022	deployed  	harbor-1.9.0	2.5.0      	Upgrade complete

So I have the following questions:

• It there a way to make this work in the current version (some configuration I have not seen)?
• Would you consider this a valuable feature for flux?
• Is there a chance that this may be added to flux? I know this would be a lot of work. Currently flux doesn't seem to care about old chart versions. The status resource of the helmchart object doesn't show a complete list of versions.

Thanks!

[stefanprodan]

You could write some CI job or GH Action that figures out which version comes next and patches the HelmRelease YAML in Git using yq.

We are currently working on OCI support for Helm and after that we have a huge backlog of things to do for Flux v2 GA. Implementing incremental upgrades would require substantial changes to the source controller reconcilers (HelmRepo, OCIRepo, HelmChart), such a major change can't happen this year.

[gprossliner]

Thanks for your fast response!

I‘m aware that I could implement this by myself. But this basically means that I would need to reimplement large amounts of the upgrade workflow like

• Repository fetch
• Version retrieval, wildcard mapping and sorting
• Version application and wait for completion

A good starting point for such an automation would be if the HelmChart status field would enumerate all version, not only the latest one matching. Then such an automation would only need to interact with the Kubernetes API.

Because flux need to fetch the Versions anyway, this should be a much smaller feature request.

Although I haven’t checked the code yet, it would be fine for me to provide a PR if wanted.

[stefanprodan]

A good starting point for such an automation would be if the HelmChart status field would enumerate all version

A chart can have numerous versions that would clog ectd or even hit the max limit of a custom resource. As for OCI, we don't know which versions are available as Flux does not stores all the container image tags.

[gprossliner]

That’s very true. I‘m really not comfortable with this too. Writing a good spec for this would also be hard - even if not considering OCI charts. I think I will not start this endeavor.

It‘s really annoying if a chart has such a principle flaw. This is not related to flux.

Because all other needed charts are working well, I think I will investigate the root problem within the specific chart, and get in touch with those guys. Maybe this could be fixed there.

A question about OCI Charts: How do you enumerate versions of charts for auto-upgrade (eg 1.*)?

Thank you.

[stefanprodan]

A question about OCI Charts: How do you enumerate versions of charts for auto-upgrade (eg 1.*)?

The Helm SDK does it for us, it uses ORAS to list tags in-memory and applies the semver filter/ordering.