RC2 draft 🎈

[todaywasawesome]

• Follow up to RC 1 draft #21
• Feedback requested this PR before the RC 2 milestone on Friday Oct 1st. Asking for feedback both on changes proposed in this PR, and on the RC 1 pre-release in general. Please use code review and general comments on this PR for feedback
• We are on currently on track to an initial full release of the principles by GitOpsCon! 🤞✨

To-do

• Bring over co-authorship participation notes from Group iteration
  • From Sept 29, 2021 - WG Fortnightly participants
• Also, add co-authorship credits to additional people who participate in this PR
• Open discussions broken out into follow-up issues:
  • Discussion on closed-loop and control theory #24

Details

For RC2 we focused on the glossary. This is actually a fairly large change to the glossary. Included in these changes are

• Removing terms that are no longer used in the principles
• Removing items that might be viewed as backdooring an opinion that is not directly supported by the principles themselves
• Cleaning up wording to be more clear
• Grammar fixes
• Removing language that might be confusing
• Reword to avoid using pronouns like "we"

Two of the larger items were to remove "Break glass" and its definition.

Break Glass
The temporary suspension of GitOps principles, often accomplished by pausing automated reconciliation. While these principles apply to typical operations, it may at times be necessary to temporarily pause reconciliation, for example during incident management activities. In these cases, other modes of operations should be considered (e.g. manual intervention), followed by any necessary updates to the desired state declarations, and finally resuming reconciliation of the system with the updated declarations. Pragmatic exceptions to these guiding principles are expected from time to time during the journey toward a system being fully managed by GitOps.

And the removal of these lines from the Software System:

One or more software repositories for storing deployable artifacts that may be loaded into the runtime environments, eg. configuration files, code, binaries, and packages
One or more Administrators who are responsible for operating the runtime environments ie. installing, starting, stopping and updating software, code, configuration, etc

In the case of break glass, this is a term that was in a previous version of the principles. The discussion of when it's ok to break glass is a large one and something that is likely better addressed in a whitepaper or a blog post. It's such an important topic that having it backended into the glossary felt like too big of an item and viewpoint to be added there.

For the software system, these lines imply that a software system includes the repo where artifacts or configurations are stored. It almost implies that if you're using Github you would need to ensure Github was operating with GitOps principles because the implication is that your repos are part of your software system. Because of this, we felt it was better to remove them.

[lloydchang]

Not now in this RC2 draft, but eventually, I would like us — this GitOps Working Group for CNCF: Application Delivery TAG — to start focusing more on Continuous Operations (CO) when our time permits. I just alluded to CO at #8 (comment).
Thank you, everyone!

[squaremo]

Practical systems needn't use triggers in this way -- it's an implementation detail. It is enough to say that reconciliation works continually to make progress towards the desired state.

[scottrigby]

@squaremo this note is here to make clear that doing CD tasks from CI systems in itself is not GitOps, because it is event-based only, not continuous. I think you're right that saying software agents work on this "continuously" implies that, but since Continuous Integration is so widely known compared to GitOps CD, I think the word "continuous" in itself needs some further clarification like this to drive home the distinction. WDYT? Is that already very clear without an additional sentence like this? Or do you think this sentence could be adjusted to make that point in a different and better way?

[lloydchang]

Suggested change

	Contrary to "CIops", any divergence between the two will trigger reconciliation, regardless of where changes occurred.
	GitOps reconciliation works continually to make progress towards the desired state. GitOps CD & CO differ from event-based operations driven by CI.

@squaremo @scottrigby @todaywasawesome WDYT?

[squaremo]

this note is here to make clear that doing CD tasks from CI systems in itself is not GitOps, because it is event-based only, not continuous.

I like @lloydchang 's suggested phrasing. I would elaborate on it a little, by saying that the reasons typical uses of CI fall short of gitops are:

• action is only taken when a change is made to the source, so it's not continually trying to make progress;
• events can be processed out of order; e.g., a pipeline fails, then is run again later and succeeds

[moshloop]

I am not sure progress is the right word here - It will most commonly be associated with progressive delivery which is outside the scope of GitOps either via each progression being a new declaration or through the abstraction of the delivery to another controller - A gitops controller shouldn't be progressively incrementing for example replicas and leaving "attempted" state in between desired and actual.

[todaywasawesome]

Closing this conversation as it's now out of date with the PR.

[scottrigby]

Reopening conversation so that if desired this can be moved to an issue for feedback between RC 2 and planned v1.0.0. Good points are made here.

[squaremo]

It's not really a case of applying the desired state. Typically, the desired state is abstract (e.g., a Kubernetes Deployment) and the observed state is concrete (these pods are running). Reconciliation takes actions to move the concrete state so that it fulfils the abstract model.

You might ask "but isn't that something Kubernetes does, rather than the GitOps system?" Kubernetes is part of the GitOps system -- you need something (e.g., Flux) to convey the abstract, desired state to Kubernetes, and Kubernetes to interpret it into a running system. Without Kubernetes doing its job, you're just pasting files into a database; and without e.g., Flux doing its job, you don't get the benefit of working via Git.

[todaywasawesome]

@squaremo I think we've closed changes to the principles themselves in #21 unless it's a major issue.

For this particular case, we used the language "attempt to apply" because it's both broader and doesn't require success. For example, if the desired state requires more nodes than the physical system has available then an attempt to apply could simply mean the software agent makes the request of Kubernetes but could include emailing and alerting an admin that they're out of nodes.

Of course, if we weren't using Kubernetes, then attempt to apply could mean a whole different set of procedures in order to attempt to apply the desired state.

[scottrigby]

@squaremo I think we've closed changes to the principles themselves in #21 unless it's a major issue.

@todaywasawesome quick note on this, I don't believe we agreed to close changes to the principles. But instead agreed to a RC/feedback approach, to give folks a chance to give feedback on RC 1, leading up to the planned RC 2 this Friday, and planned full release by Oct 11. Right?

@squaremo do you have a concrete suggestion for another non-kubernetes-specific way to phrase this other than attempting to "apply" the desired state to the actual state?

[squaremo]

do you have a concrete suggestion for another non-kubernetes-specific way to phrase this other than attempting to "apply" the desired state to the actual state?

"Taking action to make progress towards the desired state"

[scottrigby]

OK, for now changing "attempt to apply" to "attempt to reach the desired state" (simpler wording. Alternatively "attempt to make progress toward the desired state" is longer but also fine). Doing this now so that we can stay on target for publishing RC 2 today.

Because this is important, @todaywasawesome will open a discussion just on this point today, CC @squaremo, and link to this comment thread for context #22 (comment).

I believe the full "Taking action to make progress towards the desired state" is relevant to the possible 5th principle that was temporarily excluded from the RC:

5. Operated in a closed loop
   Software agents observe desired state and meta-state to take actions based on policy when the desired state cannot be reached. This may include things like notifications, rollbacks, etc.

We did not include this because it's not clear whether a closed (feedback) loop is required for GitOps. If not, it should not be a foundational principle. If so, it should. If we're on the fence about that as a group, this could all perhaps be addressed in the glossary.

So let's follow-up in that forthcoming discussion? If we can resolve that question in time, it could make it into the first full release (v1.0.0) scheduled to be ready by KubeCon NA. If not, it could be added to future releases.

[scottrigby]

To be continued in #24 😸

[scottrigby]

To everyone who may have checked out this PR locally. I have rebased this PR branch onto main and safely force pushed using force-with-lease.

I did this because this PR branch was out of date, so the file diff contained changes that are already in RC 1 and main. The file diff should be more clear now if you all want to have another look.

Thanks everyone for the feedback so far. Please keep it coming, our goal is to release RC 2 this Friday Oct 1st, so we are looking for feedback ideally with concrete suggestions.

[lloydchang]

re: #22 (comment) from @scottrigby

I just realized that I'm unable to comment via the link above, because it's from an "Outdated" commit.

I don't know if "Outdated" is an unintended side effect of the force push. Within this PR's files, the only file that I can in-line comment on is GLOSSARY.md — I cannot in-line comment on PRINCIPLES.md

I want to support @squaremo's explanation, and suggest using the phrase converge with instead of attempt to apply

WDYT @scottrigby @squaremo @todaywasawesome @williamcaban?

[lloydchang]

Suggested change

	Contrary to "CIops", any divergence between the two will trigger reconciliation, regardless of where changes occurred.
	GitOps reconciliation works continually to make progress towards the desired state. GitOps CD & CO differ from event-based operations driven by CI.

@squaremo @scottrigby @todaywasawesome WDYT?

[lloydchang]

just a note for future reference:

re: BEST_PRACTICES.md and future work... I would like us — this GitOps Working Group for CNCF: Application Delivery TAG — to start focusing more on Continuous Operations (CO) when our time permits. Thank you, everyone!

Examples in which Git is the interface to operations:

1. Application and/or Database Credentials stored in Git, then continuously operated upon by a Sealed Secrets Kubernetes controller

2. Declared Application Versions and/or corresponding Database Schemas stored in Git and/or Container Registry, then continuously operated upon by a Kubernetes init container running Liquibase

3. Git playing central role in CO is semver range we discussed at https://github.com/open-gitops/documents/pull/22/files#r718101001 in use by Flux v2 Image Automation Controller and its predecessor in Flux v1.

4. Maybe BEST-PRACTICES.md can reference vendor-neutral examples of CO in comparison to vendor-specific examples of CO described at https://www.weave.works/product/gitops-enterprise/

I feel the real power of GitOps happens with CO 24/7/365 round-the-clock, converging declared state with actual state, and beyond.

[scottrigby]

Thanks everyone for feedback on RC 1 and discussion on this PR for RC 2.

After merge, we will create an RC 2 release branch and tag the release. RC 2 is the last planned release candidate before the v1.0.0 milestone, scheduled to release before Kubecon NA, 2021.

Any open discussions from feedback will be moved to issues and attached to the appropriate milestone. These may be addressed either before the v1.0.0 milestone or after depending on how motivated its proponents are, and how consensus from the GitOps Working Group unfolds.

[scottrigby]

RC 2 tag created for another round of feedback! 🚀 🔍 https://github.com/open-gitops/documents/releases/tag/v1.0.0-rc.2

REMINDER: RC 2 is the last planned release candidate before the v1.0.0 milestone, scheduled to release by Kubecon NA, 2021. So we have 10 days now before the initial full release 🙌

Let's:

• continue discussions in issues/PRs, and attach to the v1.0.0 milestone
• share the RC 2 tag for feedback from the wider GitOps community 🙏 💬

[lloydchang]

LGTM

re: Discussion on closed-loop and control theory #24, I commented at #24 (comment)

Thanks all!