Develop policy for adopting LLVM minor version updates

[richlander]

It looks like we have an inconsistent approach to adopting LLVM/clang minor versions. We should likely revisit that.

Query: https://github.com/search?q=repo%3Adotnet%2Fdotnet-buildtools-prereqs-docker%20LLVM_VERSION&type=code

Ideally, we'd move forward with minor versions to ensure we pick up any security of performance updates. We also would need a way to validate those updates with runtime before publishing. We likely do that with some fancy (with some definition of that term) manifest.json edits to produce a new tag name for a temporary period.

@sbomer

[mthalman]

We also would need a way to validate those updates with runtime before publishing. We likely do that with some fancy (with some definition of that term) manifest.json edits to produce a new tag name for a temporary period.

Gosh, do we need a nightly/staging branch in this repo too? 🫠

[richlander]

That's a good question.

Also see #1369.

[lbussell]

I think that having two images with different floating tags for a temporary period is preferable to spinning up a second repo. Like alpine images in dotnet-docker.

[mthalman]

[Triage]

This is related to #1267 which would help to centrally manage version numbers of dependencies.

I'll log a separate issue for the aspect of allowing updated images to be available for validation before referencing them directly in a release branch: #1373

[mthalman]

[Triage]

Assigning to @sbomer to take a look at defining a policy.

[sbomer]

For the policy: I think we should stay on the latest minor/patch version whenever we can.

The challenge isn't defining the policy, it's automating it. I don't know of a good "push" mechanism to get notified about new releases, so I suggest we spin a build at least once a week that looks up the latest released minor/patch version and builds it.

edit: looks like Renovate might be a good option for the "push" option.

Validating it before publishing might be more difficult because the validation happens in a different repo. It would require a manual step in the process, or a dependency on runtime ci, which doesn't seem ideal. How do other repos handle nightly/staging rollouts?

One idea is to simply publish images with minor/patch updates automatically (no validation), and either:

• accept the instability in dotnet/runtime, using pinned images to work around build breaks, or
• prevent the dotnet/runtime instability by referencing image digests that get updated automatically.

[sbomer]

#1265 addresses the same question as this issue more generally.

[richlander]

We should bias towards stability-increasing approaches. Pinned image references enables us to merge with confidence in this repo w/o requiring a nightly branch and improves reliability in dotnet/runtime.

It hadn't occurred to me that pinned references were the solution to bumping LLVM minor versions safely. Sounds great! I was not keen on creating a whole other repo for that w/a lot care and feeding required.

[mthalman]

A pinning strategy does decrease the priority of #1373 which I'm in favor of. The use of Renovate is outlined in #1321. I'm still working through the pattern for that to reduce any security concerns.

[richlander]

Here's a thought. Let's see if you agree.

• Pinning is a generally useful strategy and is overall cheaper so we should start with it.
• If pinning isn't sufficient on its own, we should add staging to help us update images more safely.

IOW, I see these strategies more as a progression than alternatives. However, we should start with just one.

[mthalman]

If you are already pinning to a digest, I don't see there ever being a need for staging. Do you agree with that? But for others that are using floating tags, staging would be useful.

[richlander]

We want two things:

• The ability to safely make evolve existing images
• Constant flow of image rebuilds due to base image updates

With pinning, we could make a change, see a break, and then spend time to fix it, cutting off access to rebuilds for the consuming repo. We may find that there are a class of images where it takes a while to get the fix right. Staging solves that.

[mthalman]

That's fair. I agree with that. I think it comes down to confidence level. If we're reasonably confident a change won't break consuming repos but still want to be protected from breaks, let it flow via a digest update. If a broader change is being made with lower confidence on it working, then stage those changes and test them before flowing to production.