Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky test: StatefulSet created should allow to scale up after scale down to zero #4520

Open
mimowo opened this issue Mar 7, 2025 · 0 comments
Open

Flaky test: StatefulSet created should allow to scale up after scale down to zero #4520

mimowo opened this issue Mar 7, 2025 · 0 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. kind/flake Categorizes issue or PR as related to a flaky test.

Comments

@mimowo
Copy link
Contributor

mimowo commented Mar 7, 2025

/kind flake

What happened:

The test failed on unrelated branch: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/4519/pull-kueue-test-e2e-main-1-31/1898055780486090752

What you expected to happen:

no failure

How to reproduce it (as minimally and precisely as possible):

run CI

Anything else we need to know?:

End To End Suite: kindest/node:v1.31.0: [It] StatefulSet integration when StatefulSet created should allow to scale up after scale down to zero expand_less	54s
{Timed out after 45.000s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/singlecluster/statefulset_test.go:317 with:
Expected
    <int32>: 3
to be <
    <int>: 3 failed [FAILED] Timed out after 45.000s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/singlecluster/statefulset_test.go:317 with:
Expected
    <int32>: 3
to be <
    <int>: 3
In [It] at: /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/singlecluster/statefulset_test.go:319 @ 03/07/25 17:12:40.276
}
[open stderropen_in_new](https://prow.k8s.io/spyglass/lens/junit/iframe?req=%7B%22artifacts%22%3A%5B%22artifacts%2Frun-test-e2e-singlecluster-1.31.0%2Fjunit.xml%22%5D%2C%22index%22%3A2%2C%22src%22%3A%22gs%2Fkubernetes-ci-logs%2Fpr-logs%2Fpull%2Fkubernetes-sigs_kueue%2F4519%2Fpull-kueue-test-e2e-main-1-31%2F1898055780486090752%22%7D&topURL=https%3A//prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/4519/pull-kueue-test-e2e-main-1-31/1898055780486090752&lensIndex=2#)
End To End Suite: kindest/node:v1.31.0: [It] StatefulSet integration when StatefulSet created should allow to change queue name if ReadyReplicas=0 expand_less	48s
{Timed out after 45.001s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/singlecluster/statefulset_test.go:374 with:
Expected
    <int32>: 1
to equal
    <int32>: 3 failed [FAILED] Timed out after 45.001s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/singlecluster/statefulset_test.go:374 with:
Expected
    <int32>: 1
to equal
    <int32>: 3
In [It] at: /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/singlecluster/statefulset_test.go:375 @ 03/07/25 17:13:27.9
}
[open stderropen_in_new](https://prow.k8s.io/spyglass/lens/junit/iframe?req=%7B%22artifacts%22%3A%5B%22artifacts%2Frun-test-e2e-singlecluster-1.31.0%2Fjunit.xml%22%5D%2C%22index%22%3A2%2C%22src%22%3A%22gs%2Fkubernetes-ci-logs%2Fpr-logs%2Fpull%2Fkubernetes-sigs_kueue%2F4519%2Fpull-kueue-test-e2e-main-1-31%2F1898055780486090752%22%7D&topURL=https%3A//prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/4519/pull-kueue-test-e2e-main-1-31/1898055780486090752&lensIndex=2#)
End To End Suite: kindest/node:v1.31.0: [It] StatefulSet integration when StatefulSet created should delete all Pods if StatefulSet was deleted after being partially ready expand_less	53s
{Timed out after 45.001s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/singlecluster/statefulset_test.go:424 with:
Expected
    <[]v1.Pod | len:1, cap:1>: [
        {
            TypeMeta: {Kind: "", APIVersion: ""},
            ObjectMeta: {
                Name: "sts-0",
                GenerateName: "sts-",
                Namespace: "sts-e2e-zprbh",
                SelfLink: "",
                UID: "3d3510b8-1da7-4a66-939b-e7c30f87fb43",
                ResourceVersion: "3694",
                Generation: 0,
                CreationTimestamp: {
                    Time: 2025-03-07T17:13:30Z,
                },
                DeletionTimestamp: {
                    Time: 2025-03-07T17:14:06Z,
                },
                DeletionGracePeriodSeconds: 30,
                Labels: {
                    "apps.kubernetes.io/pod-index": "0",
                    "kueue.x-k8s.io/managed": "true",
                    "kueue.x-k8s.io/queue-name": "sts-lq",
                    "statefulset.kubernetes.io/pod-name": "sts-0",
                    "app": "sts-pod",
                    "controller-revision-hash": "sts-759ff9fbd8",
                    "kueue.x-k8s.io/pod-group-name": "statefulset-sts-18b6e",
                    "kueue.x-k8s.io/pod-group-pod-index": "0",
                    "kueue.x-k8s.io/podset": "5949e52e",
                },
                Annotations: {
                    "kueue.x-k8s.io/workload": "statefulset-sts-18b6e",
                    "kueue.x-k8s.io/pod-group-fast-admission": "true",
                    "kueue.x-k8s.io/pod-group-pod-index-label": "apps.kubernetes.io/pod-index",
                    "kueue.x-k8s.io/pod-group-serving": "true",
                    "kueue.x-k8s.io/pod-group-total-count": "3",
                    "kueue.x-k8s.io/pod-suspending-parent": "statefulset",
                    "kueue.x-k8s.io/role-hash": "5949e52e",
                },
                OwnerReferences: [
                    {
                        APIVersion: "apps/v1",
                        Kind: "StatefulSet",
                        Name: "sts",
                        UID: "146d532d-39ff-4e61-a377-a40cb4415419",
                        Controller: true,
                        BlockOwnerDeletion: true,
                    },
                ],
                Finalizers: nil,
                ManagedFields: [
                    {
                        Manager: "kube-controller-manager",
                        Operation: "Update",
                        APIVersion: "v1",
                        Time: {
                            Time: 2025-03-07T17:13:30Z,
                        },
                        FieldsType: "FieldsV1",
                        FieldsV1: {
                            Raw: "{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:kueue.x-k8s.io/pod-group-fast-admission\":{},\"f:kueue.x-k8s.io/pod-group-pod-index-label\":{},\"f:kueue.x-k8s.io/pod-group-serving\":{},\"f:kueue.x-k8s.io/pod-group-total-count\":{},\"f:kueue.x-k8s.io/pod-suspending-parent\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:app\":{},\"f:apps.kubernetes.io/pod-index\":{},\"f:controller-revision-hash\":{},\"f:kueue.x-k8s.io/pod-group-name\":{},\"f:kueue.x-k8s.io/queue-name\":{},\"f:statefulset.kubernetes.io/pod-name\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"146d532d-39ff-4e61-a377-a40cb4415419\\\"}\":{}}},\"f:spec\":{\"f:containers\":{\"k:{\\\"name\\\":\\\"c\\\"}\":{\".\":{},\"f:args\":{},\"f:image\":{},\"f:imagePullPolicy\":{},\"f:name\":{},\"f:resources\":{\".\":{},\"f:requests\":{\".\":{},\"f:cpu\":{}}},\"f:terminationMessagePath\":{},\"f:terminationMessagePolicy\":{}}},\"f:dnsPolicy\":{},\"f:enableServiceLinks\":{},\"f:hostname\":{},\"f:restartPolicy\":{},\"f:schedulerName\":{},\"f:securityContext\":{},\"f:terminationGracePeriodSeconds\":{}}}",
                        },
                        Subresource: "",
                    },
                    {
                        Manager: "kueue",
                        Operation: "Update",
                        APIVersion: "v1",
                        Time: {
                            Ti...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

to be empty failed [FAILED] Timed out after 45.001s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/singlecluster/statefulset_test.go:424 with:
Expected
    <[]v1.Pod | len:1, cap:1>: [
        {
            TypeMeta: {Kind: "", APIVersion: ""},
            ObjectMeta: {
                Name: "sts-0",
                GenerateName: "sts-",
                Namespace: "sts-e2e-zprbh",
                SelfLink: "",
                UID: "3d3510b8-1da7-4a66-939b-e7c30f87fb43",
                ResourceVersion: "3694",
                Generation: 0,
                CreationTimestamp: {
                    Time: 2025-03-07T17:13:30Z,
                },
                DeletionTimestamp: {
                    Time: 2025-03-07T17:14:06Z,
                },
                DeletionGracePeriodSeconds: 30,
                Labels: {
                    "apps.kubernetes.io/pod-index": "0",
                    "kueue.x-k8s.io/managed": "true",
                    "kueue.x-k8s.io/queue-name": "sts-lq",
                    "statefulset.kubernetes.io/pod-name": "sts-0",
                    "app": "sts-pod",
                    "controller-revision-hash": "sts-759ff9fbd8",
                    "kueue.x-k8s.io/pod-group-name": "statefulset-sts-18b6e",
                    "kueue.x-k8s.io/pod-group-pod-index": "0",
                    "kueue.x-k8s.io/podset": "5949e52e",
                },
                Annotations: {
                    "kueue.x-k8s.io/workload": "statefulset-sts-18b6e",
                    "kueue.x-k8s.io/pod-group-fast-admission": "true",
                    "kueue.x-k8s.io/pod-group-pod-index-label": "apps.kubernetes.io/pod-index",
                    "kueue.x-k8s.io/pod-group-serving": "true",
                    "kueue.x-k8s.io/pod-group-total-count": "3",
                    "kueue.x-k8s.io/pod-suspending-parent": "statefulset",
                    "kueue.x-k8s.io/role-hash": "5949e52e",
                },
                OwnerReferences: [
                    {
                        APIVersion: "apps/v1",
                        Kind: "StatefulSet",
                        Name: "sts",
                        UID: "146d532d-39ff-4e61-a377-a40cb4415419",
                        Controller: true,
                        BlockOwnerDeletion: true,
                    },
                ],
                Finalizers: nil,
                ManagedFields: [
                    {
                        Manager: "kube-controller-manager",
                        Operation: "Update",
                        APIVersion: "v1",
                        Time: {
                            Time: 2025-03-07T17:13:30Z,
                        },
                        FieldsType: "FieldsV1",
                        FieldsV1: {
                            Raw: "{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:kueue.x-k8s.io/pod-group-fast-admission\":{},\"f:kueue.x-k8s.io/pod-group-pod-index-label\":{},\"f:kueue.x-k8s.io/pod-group-serving\":{},\"f:kueue.x-k8s.io/pod-group-total-count\":{},\"f:kueue.x-k8s.io/pod-suspending-parent\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:app\":{},\"f:apps.kubernetes.io/pod-index\":{},\"f:controller-revision-hash\":{},\"f:kueue.x-k8s.io/pod-group-name\":{},\"f:kueue.x-k8s.io/queue-name\":{},\"f:statefulset.kubernetes.io/pod-name\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"146d532d-39ff-4e61-a377-a40cb4415419\\\"}\":{}}},\"f:spec\":{\"f:containers\":{\"k:{\\\"name\\\":\\\"c\\\"}\":{\".\":{},\"f:args\":{},\"f:image\":{},\"f:imagePullPolicy\":{},\"f:name\":{},\"f:resources\":{\".\":{},\"f:requests\":{\".\":{},\"f:cpu\":{}}},\"f:terminationMessagePath\":{},\"f:terminationMessagePolicy\":{}}},\"f:dnsPolicy\":{},\"f:enableServiceLinks\":{},\"f:hostname\":{},\"f:restartPolicy\":{},\"f:schedulerName\":{},\"f:securityContext\":{},\"f:terminationGracePeriodSeconds\":{}}}",
                        },
                        Subresource: "",
                    },
                    {
                        Manager: "kueue",
                        Operation: "Update",
                        APIVersion: "v1",
                        Time: {
                            Ti...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

to be empty
In [It] at: /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/singlecluster/statefulset_test.go:425 @ 03/07/25 17:14:21.478
}
@mimowo mimowo added the kind/bug Categorizes issue or PR as related to a bug. label Mar 7, 2025
@k8s-ci-robot k8s-ci-robot added the kind/flake Categorizes issue or PR as related to a flaky test. label Mar 7, 2025
@mimowo mimowo mentioned this issue Mar 7, 2025
Open
@tenzen-y tenzen-y mentioned this issue Mar 8, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. kind/flake Categorizes issue or PR as related to a flaky test.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mimowo @k8s-ci-robot