Stop supporting FlowSchema and PriorityLevelConfiguration v1beta3.

[mbobrovskyi]

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

Merge the FlowSchema and PriorityLevelConfiguration v1 manifests into the installation manifests and remove v1beta3 manifests.

Which issue(s) this PR fixes:

Part of #3046

Special notes for your reviewer:

Does this PR introduce a user-facing change?

The APF configuration, using v1beta3 API dedicated for Kubernetes 1.28 (or older), for the visibility server is no longer part of the release artifacts of Kueue.

[netlify]

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	eee2e4c
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/6798ae012b5569000869edf2

[mbobrovskyi]

/hold for v0.10.1 release

[mimowo]

Thanks for the PR, I would like to make sure we are safe to drop the support.

IIUC the plan is to only install v1 instead of v1beta3? In that case, what is the first version of k8s supporting v1?

[mbobrovskyi]

Thanks for the PR, I would like to make sure we are safe to drop the support.

IIUC the plan is to only install v1 instead of v1beta3? In that case, what is the first version of k8s supporting v1?

The documentation states that API Priority and Fairness has been stable since Kubernetes v1.29.

[mimowo]

Was v1 supported prior to the graduation?

[mbobrovskyi]

Was v1 supported prior to the graduation?

As I can see v1 support from v1.29 (see here).

[tenzen-y]

This indicates that we will change the implied supporting kubernetes version to v1.29 (https://github.com/kubernetes-sigs/kueue?tab=readme-ov-file#installation).

I'm ok since the v1.28 is no longer supported in OSS kubernetes. But let me confirm if @mimowo agrees with the changes.

[mimowo]

TBH, I'm not sure, the reason is that cloud providers give usually a little longer support than OSS, and so it is common to see users on 1.28.

For example the policy for gke is 14months of support which is 2 months longer than oss: https://cloud.google.com/kubernetes-engine/versioning (and so it is possible to create new clusters on 1.28 on gke using standard release channels).

I think that as long as it is not extra difficult we try to support one version beyond EOL, but I would like to hear also from @kannon92 and @dgrove-oss

EDIT: my preference is to officially support in Kueue non-EOL for Kubernetes, but when possible and not too difficult then try to support one release beyond. So, in this case I would wait until 1.29 is EOL, but this is not a strong view.

[mimowo]

@mbobrovskyi if we merge this PR would Kueue be installed and working on k8s 1.28 for the base use cases? or it will be no longer usable even for simple use cases? Can you maybe check on a running k8s 1.28 manually?

I think we definitely want to be able to still install Kueue and support simple use-cases for older versions. It is ok, if some features like pod integration don't work on 1.28, but the very basic features I expect need to work on 1 or even 2 releases beyond EOL.

[mbobrovskyi]

@mbobrovskyi if we merge this PR would Kueue be installed and working on k8s 1.28 for the base use cases? or it will be no longer usable even for simple use cases? Can you maybe check on a running k8s 1.28 manually?

I think we definitely want to be able to still install Kueue and support simple use-cases for older versions. It is ok, if some features like pod integration don't work on 1.28, but the very basic features I expect need to work on 1 or even 2 releases beyond EOL.

This PR added FlowSchema and PriorityLevelConfiguration with version v1 to the main manifest. However, Kubernetes 1.28 does not support the v1 version. Therefore, users must manually change the version of these resources to v1beta3.

[mimowo]

What happens if the users without any manual changes just follow the installation steps from the documentation https://kueue.sigs.k8s.io/docs/installation/#install-a-released-version? Will the installation break, or how the users would get see that it is not working?

[mbobrovskyi]

What happens if the users without any manual changes just follow the installation steps from the documentation https://kueue.sigs.k8s.io/docs/installation/#install-a-released-version? Will the installation break, or how the users would get see that it is not working?

k apply -f artifacts/manifests.yaml --server-side
...
resource mapping not found for name: "kueue-visibility" namespace: "kueue-system" from "artifacts/manifests.yaml": no matches for kind "FlowSchema" in version "flowcontrol.apiserver.k8s.io/v1"
ensure CRDs are installed first
resource mapping not found for name: "kueue-visibility" namespace: "kueue-system" from "artifacts/manifests.yaml": no matches for kind "PriorityLevelConfiguration" in version "flowcontrol.apiserver.k8s.io/v1"
ensure CRDs are installed first
...

[mimowo]

I think that if a vendor chooses to support beyond EOL it is on them to support that. Wearing my Red Hat it
would be great to have LTS on old release branches but it opens up a large can of worms for upstream projects to
handle.

Right, I don't want to have a rule on supporting more than non-EOL k8s, just best effort. With this PR merged Kueue would not even install on 1.28. So, the installation guide would be wrong: https://github.com/kubernetes-sigs/kueue?tab=readme-ov-file#installation as we say "Requires Kubernetes 1.25 or newer.".

So, I think while it is ok to break older version a little bit on some features, not being even able to install is bad.
So the options I see:

1. update the installation guide saying "Requires Kubernetes 1.29 or newer.", but this is a very restricting claim I think for now
2. add a section in the installation guide for users on < 1.29
3. wait a bit more, I propose 1 release, merge the PR then and update the installation guide saying "Requires Kubernetes 1.29 or newer."

[mimowo]

Maybe there is another possibility to drop generation of the artifact for 1.28, but still keep the visibility-apf for v1 as opt-in for the foreseeable future, WDYT?

[mbobrovskyi]

Maybe there is another possibility to drop generation of the artifact for 1.28, but still keep the visibility-apf for v1 as opt-in for the foreseeable future, WDYT?

SGTM

[mimowo]

ack, added release note

[mimowo]

/hold
To clarify the support requirements and consequences of the PR for the support in https://github.com/kubernetes-sigs/kueue/pull/3983/files/646addabe7999f0126bac4d19e2ba0b77d81bada#r1925026803

[mimowo]

/release-note-edit

The APF configuration, using v1beta3 API dedicated for Kubernetes 1.28 (or older), for the visibility server is no longer part of the release artifacts of Kueue. 

[mimowo]

/lgtm
/approve

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 940109af67e3a4cb0e8c3bef521bacb8e35eca7c

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mbobrovskyi, mimowo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mimowo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mimowo]

/hold cancel
The recent update only drops support for 1.28 (which is EOL), but keeps the APF opt-in, which will allow to install Kueue 0.11 on older versions of Kubernetes, which was my main concern in #3983 (comment)