Ensure commit was already once on the cluster before rolling out next commit

[rupelu]

Lets assume we have two clusters A and B. Both are getting configuration from a single git repository. The main GitRepository resource is configured to be a tag. Sometimes cluster B is down for a longer period of time, meanwhile the flux configuration changes multiple times. Cluster A gets all the updates one by one, while B might jump over some commits. This can be dangerous, if some of the jumped commits are migration steps, like suspend/resume resources, creating/deleting migration jobs, enable/disable pruning.

Below is an example commit timeline. In this timeline commit 2 creates a new database and starts a migration job. The old database is then deleted in commit 3. It would be fatal for a cluster to jump from commit 1 to commit 3 or beyond.

         mandatory  commits/tags
            ┌────────────────┐
            │                │
      ┌──┐     ┌──┐      ┌──┐       ┌──┐
 ---──┤ 1├─────┤ 2├──────┤ 3├───────┤ 4├---
      └──┘     └──┘      └──┘       └──┘
       |        |          |
       |        |          |
       |        |    - delete old db
simple commit   |    - remove migration job
                |
            - create db
            - create migration job
              to move data from old db to new db

So what I am looking for is a way to define mandatory tags for a GitRepository before rolling out the next tag. When a mandatory tag is rolled out, all Kustomizations referencing the repository need to fulfill their health checks to continue to the next tag.
Currently I have no good idea on how to mark a tag as mandatory or how to tell flux that there were mandatory tags in between. Maybe the the GitRepository would need some kind of policy for mandatory tags, that tells to only do minor updates and major updates only from tags which are referenced by the policy. Which could look maybe like this:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  name: podinfo
  namespace: default
spec:
  interval: 5m0s
  url: https://github.com/stefanprodan/podinfo
  ref:
    tag: 2.4.3                # Lets assume the previously applied tag would be 2.2.x so we would need to 
                              # make sure 2.3.x is applied before the target tag is applied.
    # -- OR we could specify a range with semver (as far as I know this does also not guarantee by itself which tags will be fully applied on the cluster)
    semver: ">= 1.0.0, < 4"   # Here semver is an example to illustrate the possible version range
  policy:
    tag:
      minStep: incremental   # it is ok to increment when a new version is there
      maxStep: minor         # maximum step to take should be minor version upgrade
      jumps:                 # when 'from' version is the current version, allow to jump to defined version
      - from: 1.6.3
        to: 2.0.0
      - from: 2.4.3
        to: 3.0.0

Somehow the source-controller would need to check:

• the currently applied tag
• the tags available on the remote
• next tag to select based on the policy

With that we could:

• ensure that a pair of commits will run one after the other (see example)
• automate the moving of resources as stated in the Flux FAQ
• enforce a migration path policy (only migrate to the next mandatory commit/tag)

I have no idea whether flux is seen as a tool to rollout migration tasks over git at all. We could of course make these migrations manually but that would again require to touch the clusters with kubectl - which should not be needed when managing migrations in a GitOps way.

[hendrikhalkow]

Let's reconsider what Flux and GitOps actually are: The idea is that a single commit inside a Git repository describes the desired state. It is the reconciler's job to make sure that the reality matches the desired state. It's also the reconciler's job to find out what's necessary to bring a system into that desired state.

What you propose is that a series of commits, a branch, contains a series of commands that are different in each commit that need to be executed in order to bring a system into a desired state. This is not GitOps, because the desired state is not defined by a single commit.

But here is the good news: For database migrations, two production-proven tools, Flyway and Liquibase exist that perfectly solve that problem. They keep a ledger of database migration steps that are part of the application. The full list of required migration steps in in every commit, so no need to "execute" commits in order.

[makkes]

Agreed 100%. Plus:

This can be dangerous, if some of the jumped commits are migration steps

1. How would you go about setting up a new cluster from a git repository with thousands of commits? Setting that up by applying all commits in sequence would take ages.
2. How would someone looking at the repository at a certain point in time understand this sequence without having to dig into its history?
3. What if some manifest files have been upgraded to a newer API version that is removed in current Kubernetes versions? Applying all commits in sequence would lead to these old versions being applied without success because the API version wouldn't be supported, anymore.

Each commit in your repository should denote a desired state. If a certain file is removed by a particular commit, then the desired state is represented by all remaining files and the removed one should not matter, anymore.

You may want to revisit your repository structure to match the GitOps principles.