Enable sorting of queue items based on a custom timestamp extracted from an annotation

[gbenhaim]

What would you like to be added:

Enable sorting of queue items based on a custom timestamp extracted from an annotation.

Why is this needed:

Queuing plain pods created by Tekton can be highly beneficial for sharing quotas and achieving more consistent scheduling (FIFO based on the pod's creation timestamp). A short demo on this topic can be found here.

The experience could be further improved by allowing pods in the queue to be sorted based on the creation timestamp of the Tekton PipelineRun they belong to.

This would prioritize pods associated with "older" pipelines, helping them complete first and reducing timeouts.

Completion requirements:

1. Kueue should be able to read a timestamp from an annotation set on the workload and use it for sorting workloads in the queue.

2. It should be possible to copy annotations from the Job to the Workload, similar to how labels can be copied.

3. Tekton should set the creation timestamp of the PipelineRun as an annotation on the pods belonging to that PipelineRun (this should be implemented in Tekton).

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[kannon92]

I am a bit confused.

https://cloud.google.com/kubernetes-engine/docs/tutorials/kueue-intro

This seems to say that we have support of FIFO for queueing.

[kannon92]

https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/#queueing-strategy

[tenzen-y]

We have received similar requests many times. However, we could not accept it since such custom timestamp could violate fairness. All workloads should be fair under the Kueue algorithms.

Such annotation allows evil users to prioritize their tasks.

[gbenhaim]

@tenzen-y thank you for your response. The feature I'm suggesting can be disabled by default and enabled by an administrator that would like it. The annotation on the Job which contains the cusotm timestamp, for example, can be protected by an admission web hook that will allow only specific controllers to set it or by using RBAC. Another example, in a multi tenant scenario, if each tenant has its own ClusterQueue, tenants can't prioritize their Workloads over other other tenants.

I think that the fact there were several similar requests is an indication for a real use case.

[tenzen-y]

The annotation on the Job which contains the cusotm timestamp, for example, can be protected by an admission web hook that will allow only specific controllers to set it or by using RBAC.

I think the current priority and creation timestamp base queuing is a guardrail to keep consistency across all Kueue features.
If we allow them to disable it by this feature request, we will probably lose stability based on both development and User PoV.

Another example, in a multi tenant scenario, if each tenant has its own ClusterQueue, tenants can't prioritize their Workloads over other other tenants.

Is there any reason why WorkloadPriorityClass and core PriorityClass do not satisfy this case?