fix(sdk): Add Kubernetes refs to the JobSet OpenAPI swagger

[andreyvelich]

What type of PR is this?

/kind bug

What this PR does / why we need it:

After we migrated to the latest OpenAPI generator: openapi-generator-cli:v7.11.0, it generates JobSet models using the Pydantic library. It requires for all external models to be presented, otherwise serialization will fail:

from_dict()
to_dict()

As you can see, the JobsetV1alpha2JobSet class depends on V1ObjectMeta class which we don't import in the jobset_v1alpha2_job_set.py file.

To solve this problem, I've updated JobSet swagger to include external Kubernetes models, for instance

"$ref": "https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/v1.32.2/api/openapi-spec/swagger.json#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta"

cc @tenzen-y @kannon92 @ahg-g @Electronic-Waste @danielvegamyhre @epicseven-cup @astefanutti

Special notes for your reviewer:

We need this fix for Kubeflow Trainer integration: kubeflow/trainer#2466

Does this PR introduce a user-facing change?

NONE

[k8s-ci-robot]

Hi @andreyvelich. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[netlify]

✅ Deploy Preview for kubernetes-sigs-jobset ready!

Name	Link
🔨 Latest commit	70cb4cc
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-jobset/deploys/67c61fd47c662b0008ac35f9
😎 Deploy Preview	https://deploy-preview-810--kubernetes-sigs-jobset.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[ahg-g]

/ok-to-test

[ahg-g]

Is this embedding all jobset api dependencies in jobset's python sdk? is this a common practice?

[kannon92]

/hold

@andreyvelich is working on this for kubeflow. I don't have a strong stance on this but I'd like to get buy in from kubeflow's PR until we merge this one.

[andreyvelich]

Is this embedding all jobset api dependencies in jobset's python sdk?

Yes, otherwise the serialization will fail.
As I said, currently JobSet models uses Kubernetes models that don't exist in JobSet SDK package: https://github.com/kubernetes-sigs/jobset/blob/main/sdk/python/jobset/models/jobset_v1alpha2_replicated_job.py#L33.

is this a common practice?

I noticed that Argo Workflows also import all Kubernetes dependencies in their SDK models: https://github.com/argoproj/argo-workflows/blob/main/sdks/python/client/argo_workflows/model/object_meta.py.

Argo Workflows has the valid OpenAPI spec swagger with all definitions: https://github.com/argoproj/argo-workflows/blob/main/api/openapi-spec/swagger.json

[andreyvelich]

@kannon92 @ahg-g FYI, this fix also blocks us to use v0.8.0 JobSet Python package in our Kubeflow SDK.
I had to downgrade the JobSet version to avoid error in e2e: https://github.com/kubeflow/trainer/pull/2470/files#diff-271810b5660339392a509b5e1e83a9272497a323dabc5eb82910292c2f4ce87aR34-R36

The same error user reported in Slack:
https://cloud-native.slack.com/archives/C0742LDFZ4K/p1741077959344629?thread_ts=1740708722.833979&cid=C0742LDFZ4K

[andreyvelich]

Do we want to consider to cherry-pick it for v0.8.1 if that makes sense?

[ahg-g]

Do we want to consider to cherry-pick it for v0.8.1 if that makes sense?

I am ok to cherry-pick if this unblocks kubeflow trainer

[kannon92]

/lgtm

Have we got consensus on the KubeFlow PR for this?

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich, kannon92

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [kannon92]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[andreyvelich]

Have we got consensus on the KubeFlow PR for this?

I think, so.
@tenzen-y @Electronic-Waste @astefanutti @saileshd1402 @deepanker13 @shravan-achar @akshaychitneni Any concerns to merge this PR?

[Electronic-Waste]

I'm ok for this PR:)

[tenzen-y]

/lgtm

Thank you.

[tenzen-y]

/lgtm

Have we got consensus on the KubeFlow PR for this?

Based on this comment.
/hold cancel

[kannon92]

/cherry-pick release-0.8

[tenzen-y]

/cherry-pick release-0.8

[kannon92]

@tenzen-y what are we missing to do this? Some kind of test-infra plugin?

[tenzen-y]

@tenzen-y what are we missing to do this? Some kind of test-infra plugin?

This one: kubernetes/test-infra#34463

[tenzen-y]

/cherry-pick release-0.8

[k8s-infra-cherrypick-robot]

@tenzen-y: Failed to get PR patch from GitHub. This PR will need to be manually cherrypicked.

Error message

status code 406 not one of [200], body: {"message":"Sorry, the diff exceeded the maximum number of lines (20000)","errors":[{"resource":"PullRequest","field":"diff","code":"too_large"}],"documentation_url":"https://docs.github.com/rest/pulls/pulls#get-a-pull-request","status":"406"}

In response to this:

/cherry-pick release-0.8

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[tenzen-y]

status code 406 not one of [200], body: {"message":"Sorry, the diff exceeded the maximum number of lines (20000)","errors":[{"resource":"PullRequest","field":"diff","code":"too_large"}],"documentation_url":"https://docs.github.com/rest/pulls/pulls#get-a-pull-request","status":"406"}

It seems that diff size exceeds the limits 😓

[kannon92]

@andreyvelich can you create a manually cherry-pick for this into 0.8?