Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(sdk): Generate external Kubernetes and JobSet models #2466

Merged
merged 8 commits into from
Mar 5, 2025
Merged

feat(sdk): Generate external Kubernetes and JobSet models #2466

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
f844687
feat(sdk): Generate external Kubernetes and JobSet models
andreyvelich Mar 5, 2025
c947d08
Update JobSet commit in Swagger
andreyvelich Mar 5, 2025
d161638
Fix formats for resources and num proc
andreyvelich Mar 5, 2025
0827fce
Add custom Kind config
andreyvelich Mar 5, 2025
330ae40
Use ginkgo to run e2e
andreyvelich Mar 5, 2025
bbe2d4b
Revert Kind config change
andreyvelich Mar 5, 2025
76e4b96
Fix ginkgo binary
andreyvelich Mar 5, 2025
b5b4597
Print controller logs in case of failure
andreyvelich Mar 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,8 @@ exclude: |
(?x)^(
docs/images/.*|
pkg/client/.*|
sdk/kubeflow/trainer/[^/]*.py|
sdk/kubeflow/trainer/__init__.py|
sdk/kubeflow/trainer/api/__init__.py|
sdk/kubeflow/trainer/models/.*|
sdk/docs/.*
)$
10 changes: 8 additions & 2 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,11 +36,17 @@ PROJECT_DIR := $(shell dirname $(abspath $(lastword $(MAKEFILE_LIST))))
# Tool Binaries
LOCALBIN ?= $(PROJECT_DIR)/bin

CONTROLLER_GEN ?= $(LOCALBIN)/controller-gen
GINKGO ?= $(LOCALBIN)/ginkgo
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tenzen-y I noticed a new error in our e2e:

Will run 1 of 1 specs
panic: test timed out after 10m0s
	running tests:
		TestAPIs (10m0s)

goroutine 2578 [running]:
testing.(*M).startAlarm.func1()
	/opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:2373 +0x385
created by time.goFunc
	/opt/hostedtoolcache/go/1.23.0/x64/src/time/sleep.go:215 +0x2d

goroutine 1 [chan receive, 10 minutes]:
testing.(*T).Run(0xc0000f44e0, {0x1999495?, 0x0?}, 0x1aaae28)
	/opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:1751 +0x3ab
testing.runTests.func1(0xc0000f44e0)
	/opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:2168 +0x37
testing.tRunner(0xc0000f44e0, 0xc0000a9c70)
	/opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:1690 +0xf4
testing.runTests(0xc00038e2b8, {0x29962d0, 0x1, 0x1}, {0x4[74](https://github.com/kubeflow/trainer/actions/runs/13681536090/job/38254906117?pr=2466#step:7:75)a25?, 0xc000687350?, 0x29de060?})
	/opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:2166 +0x43d
testing.(*M).Run(0xc0002aae60)
	/opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:2034 +0x64a
main.main()
	_testmain.go:45 +0x9b

goroutine 72 [select, 10 minutes]:
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode(_, {0x6, 0x4, {0x19e625f, 0x35}, 0xc0006942a0, {{0x200d249, 0x59}, 0x2d, {0x0, ...}, ...}, ...}, ...)
	/home/runner/work/trainer/trainer/go/pkg/mod/github.com/onsi/ginkgo/[email protected]/internal/suite.go:916 +0xfc5
github.com/onsi/ginkgo/v2/internal.(*group).attemptSpec(0xc000169810, 0x1, {{0xc0001bd208?, 0xc000251b90?, 0x2?}, 0x0?})
	/home/runner/work/trainer/trainer/go/pkg/mod/github.com/onsi/ginkgo/[email protected]/internal/group.go:199 +0xc33
github.com/onsi/ginkgo/v2/internal.(*group).run(0xc000169810, {0xc000365280?, 0x0?, 0x0?})
	/home/runner/work/trainer/trainer/go/pkg/mod/github.com/onsi/ginkgo/[email protected]/internal/group.go:349 +0x905
github.com/onsi/ginkgo/v2/internal.(*Suite).runSpecs(0xc0001d2a88, {0x19b143a, 0x1a}, {0x2a009c0, 0x0, 0x0}, {0xc000050064, 0x4d}, 0x0, {0xc0001460a0, ...})
	/home/runner/work/trainer/trainer/go/pkg/mod/github.com/onsi/ginkgo/[email protected]/internal/suite.go:489 +0xa9b
github.com/onsi/ginkgo/v2/internal.(*Suite).Run(_, {_, _}, {_, _, _}, {_, _}, _, {0x1c58f30, ...}, ...)
	/home/runner/work/trainer/trainer/go/pkg/mod/github.com/onsi/ginkgo/[email protected]/internal/suite.go:130 +0x405
github.com/onsi/ginkgo/v2.RunSpecs({0x1c363c0, 0xc00026[78](https://github.com/kubeflow/trainer/actions/runs/13681536090/job/38254906117?pr=2466#step:7:79)60}, {0x19b143a, 0x1a}, {0x0, 0x0, 0x0})
	/home/runner/work/trainer/trainer/go/pkg/mod/github.com/onsi/ginkgo/[email protected]/core_dsl.go:300 +0x893
github.com/kubeflow/trainer/test/e2e.TestAPIs(0xc000267860)
	/home/runner/work/trainer/trainer/go/src/github.com/kubeflow/trainer/test/e2e/suite_test.go:39 +0x45
testing.tRunner(0xc000267[86](https://github.com/kubeflow/trainer/actions/runs/13681536090/job/38254906117?pr=2466#step:7:87)0, 0x1aaae28)
	/opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:1690 +0xf4
created by testing.(*T).Run in goroutine 1
	/opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:1743 +0x390

Since we use ginkgo to run our tests, can we use their binary to run tests similar to JobSet and Kueue?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using ginkgo binary would be more better

ENVTEST ?= $(LOCALBIN)/setup-envtest
CONTROLLER_GEN ?= $(LOCALBIN)/controller-gen
KIND ?= $(LOCALBIN)/kind

# Instructions to download tools for development.

.PHONY: ginkgo
ginkgo: ## Download the ginkgo binary if required.
GOBIN=$(LOCALBIN) go install github.com/onsi/ginkgo/v2/ginkgo@$(shell go list -m -f '{{.Version}}' github.com/onsi/ginkgo/v2)

.PHONY: envtest
envtest: ## Download the setup-envtest binary if required.
GOBIN=$(LOCALBIN) go install sigs.k8s.io/controller-runtime/tools/[email protected]
Expand Down Expand Up @@ -139,7 +145,7 @@ test-e2e-setup-cluster: kind ## Setup Kind cluster for e2e test.

.PHONY: test-e2e
test-e2e: ## Run Go e2e test.
go test ./test/e2e/...
$(GINKGO) -v ./test/e2e/...

# Input and output location for Notebooks executed with Papermill.
NOTEBOOK_INPUT=$(PROJECT_DIR)/examples/pytorch/image-classification/mnist.ipynb
Expand Down
50 changes: 25 additions & 25 deletions api/openapi-spec/swagger.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
"metadata": {
"description": "Standard object's metadata.",
"default": {},
"$ref": "#/definitions/v1.ObjectMeta"
"$ref": "https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/v1.32.2/api/openapi-spec/swagger.json#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta"
},
"spec": {
"description": "Specification of the desired ClusterTrainingRuntime.",
Expand Down Expand Up @@ -55,7 +55,7 @@
"metadata": {
"description": "Standard list metadata.",
"default": {},
"$ref": "#/definitions/v1.ListMeta"
"$ref": "https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/v1.32.2/api/openapi-spec/swagger.json#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta"
}
}
},
Expand Down Expand Up @@ -89,7 +89,7 @@
"type": "array",
"items": {
"default": {},
"$ref": "#/definitions/v1.EnvVar"
"$ref": "https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/v1.32.2/api/openapi-spec/swagger.json#/definitions/io.k8s.api.core.v1.EnvVar"
},
"x-kubernetes-list-map-keys": [
"name"
Expand All @@ -101,7 +101,7 @@
"type": "array",
"items": {
"default": {},
"$ref": "#/definitions/v1.EnvFromSource"
"$ref": "https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/v1.32.2/api/openapi-spec/swagger.json#/definitions/io.k8s.api.core.v1.EnvFromSource"
},
"x-kubernetes-list-type": "atomic"
},
Expand All @@ -115,7 +115,7 @@
"type": "array",
"items": {
"default": {},
"$ref": "#/definitions/v1.VolumeMount"
"$ref": "https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/v1.32.2/api/openapi-spec/swagger.json#/definitions/io.k8s.api.core.v1.VolumeMount"
},
"x-kubernetes-list-map-keys": [
"name"
Expand Down Expand Up @@ -144,7 +144,7 @@
"type": "array",
"items": {
"default": {},
"$ref": "#/definitions/v1.EnvVar"
"$ref": "https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/v1.32.2/api/openapi-spec/swagger.json#/definitions/io.k8s.api.core.v1.EnvVar"
},
"x-kubernetes-list-map-keys": [
"name"
Expand All @@ -153,7 +153,7 @@
},
"secretRef": {
"description": "Reference to the secret with credentials to download dataset. Secret must be created in the TrainJob's namespace.",
"$ref": "#/definitions/v1.LocalObjectReference"
"$ref": "https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/v1.32.2/api/openapi-spec/swagger.json#/definitions/io.k8s.api.core.v1.LocalObjectReference"
},
"storageUri": {
"description": "Storage uri for the dataset provider.",
Expand All @@ -170,7 +170,7 @@
"type": "array",
"items": {
"default": {},
"$ref": "#/definitions/v1.EnvVar"
"$ref": "https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/v1.32.2/api/openapi-spec/swagger.json#/definitions/io.k8s.api.core.v1.EnvVar"
},
"x-kubernetes-list-map-keys": [
"name"
Expand All @@ -179,7 +179,7 @@
},
"secretRef": {
"description": "Reference to the secret with credentials to download model. Secret must be created in the TrainJob's namespace.",
"$ref": "#/definitions/v1.LocalObjectReference"
"$ref": "https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/v1.32.2/api/openapi-spec/swagger.json#/definitions/io.k8s.api.core.v1.LocalObjectReference"
},
"storageUri": {
"description": "Storage uri for the model provider.",
Expand All @@ -194,12 +194,12 @@
"metadata": {
"description": "Metadata for custom JobSet's labels and annotations. JobSet name and namespace is equal to the TrainJob's name and namespace.",
"default": {},
"$ref": "#/definitions/v1.ObjectMeta"
"$ref": "https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/v1.32.2/api/openapi-spec/swagger.json#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta"
},
"spec": {
"description": "Specification of the desired JobSet which will be created from TrainJob.",
"default": {},
"$ref": "#/definitions/jobset.v1alpha2.JobSetSpec"
"$ref": "https://raw.githubusercontent.com/kubernetes-sigs/jobset/d5c7bcebe739a4577e30944370c2d7a68321a929/hack/python-sdk/swagger.json#/definitions/jobset.v1alpha2.JobSetSpec"
}
}
},
Expand Down Expand Up @@ -330,7 +330,7 @@
"type": "array",
"items": {
"default": {},
"$ref": "#/definitions/v1.EnvVar"
"$ref": "https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/v1.32.2/api/openapi-spec/swagger.json#/definitions/io.k8s.api.core.v1.EnvVar"
},
"x-kubernetes-list-map-keys": [
"name"
Expand All @@ -339,7 +339,7 @@
},
"secretRef": {
"description": "Reference to the secret with credentials to export model. Secret must be created in the TrainJob's namespace.",
"$ref": "#/definitions/v1.LocalObjectReference"
"$ref": "https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/v1.32.2/api/openapi-spec/swagger.json#/definitions/io.k8s.api.core.v1.LocalObjectReference"
},
"storageUri": {
"description": "Storage uri for the model exporter.",
Expand Down Expand Up @@ -424,7 +424,7 @@
"type": "array",
"items": {
"default": {},
"$ref": "#/definitions/v1.Toleration"
"$ref": "https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/v1.32.2/api/openapi-spec/swagger.json#/definitions/io.k8s.api.core.v1.Toleration"
},
"x-kubernetes-list-type": "atomic"
},
Expand All @@ -433,7 +433,7 @@
"type": "array",
"items": {
"default": {},
"$ref": "#/definitions/v1.Volume"
"$ref": "https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/v1.32.2/api/openapi-spec/swagger.json#/definitions/io.k8s.api.core.v1.Volume"
},
"x-kubernetes-list-map-keys": [
"name"
Expand Down Expand Up @@ -496,7 +496,7 @@
"type": "array",
"items": {
"default": {},
"$ref": "#/definitions/k8s.io.api.autoscaling.v2.MetricSpec"
"$ref": "https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/v1.32.2/api/openapi-spec/swagger.json#/definitions/io.k8s.api.autoscaling.v2.MetricSpec"
},
"x-kubernetes-list-type": "atomic"
},
Expand All @@ -517,7 +517,7 @@
},
"numProcPerNode": {
"description": "Number of processes per node. This value is inserted into the `--nproc-per-node` argument of the `torchrun` CLI. Supported values: `auto`, `cpu`, `gpu`, or int value. Defaults to `auto`.",
"$ref": "#/definitions/k8s.io.apimachinery.pkg.util.intstr.IntOrString"
"$ref": "https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/v1.32.2/api/openapi-spec/swagger.json#/definitions/io.k8s.apimachinery.pkg.util.intstr.IntOrString"
}
}
},
Expand All @@ -536,7 +536,7 @@
"metadata": {
"description": "Standard object's metadata.",
"default": {},
"$ref": "#/definitions/v1.ObjectMeta"
"$ref": "https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/v1.32.2/api/openapi-spec/swagger.json#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta"
},
"spec": {
"description": "Specification of the desired TrainJob.",
Expand Down Expand Up @@ -576,7 +576,7 @@
"metadata": {
"description": "Standard list metadata.",
"default": {},
"$ref": "#/definitions/v1.ListMeta"
"$ref": "https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/v1.32.2/api/openapi-spec/swagger.json#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta"
}
}
},
Expand Down Expand Up @@ -648,7 +648,7 @@
"type": "array",
"items": {
"default": {},
"$ref": "#/definitions/v1.Condition"
"$ref": "https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/v1.32.2/api/openapi-spec/swagger.json#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Condition"
},
"x-kubernetes-list-map-keys": [
"type"
Expand Down Expand Up @@ -698,7 +698,7 @@
"type": "array",
"items": {
"default": {},
"$ref": "#/definitions/v1.EnvVar"
"$ref": "https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/v1.32.2/api/openapi-spec/swagger.json#/definitions/io.k8s.api.core.v1.EnvVar"
},
"x-kubernetes-list-map-keys": [
"name"
Expand All @@ -716,11 +716,11 @@
},
"numProcPerNode": {
"description": "Number of processes/workers/slots on every training node. For the Torch runtime: `auto`, `cpu`, `gpu`, or int value can be set. For the MPI runtime only int value can be set.",
"$ref": "#/definitions/k8s.io.apimachinery.pkg.util.intstr.IntOrString"
"$ref": "https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/v1.32.2/api/openapi-spec/swagger.json#/definitions/io.k8s.apimachinery.pkg.util.intstr.IntOrString"
},
"resourcesPerNode": {
"description": "Compute resources for each training node.",
"$ref": "#/definitions/v1.ResourceRequirements"
"$ref": "https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/v1.32.2/api/openapi-spec/swagger.json#/definitions/io.k8s.api.core.v1.ResourceRequirements"
}
}
},
Expand All @@ -739,7 +739,7 @@
"metadata": {
"description": "Standard object's metadata.",
"default": {},
"$ref": "#/definitions/v1.ObjectMeta"
"$ref": "https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/v1.32.2/api/openapi-spec/swagger.json#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta"
},
"spec": {
"description": "Specification of the desired TrainingRuntime.",
Expand Down Expand Up @@ -774,7 +774,7 @@
"metadata": {
"description": "Standard list metadata.",
"default": {},
"$ref": "#/definitions/v1.ListMeta"
"$ref": "https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/v1.32.2/api/openapi-spec/swagger.json#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta"
}
}
},
Expand Down
9 changes: 8 additions & 1 deletion hack/e2e-setup-cluster.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,14 @@ cd manifests/overlays/manager
kustomize edit set image kubeflow/trainer-controller-manager=${CONTROLLER_MANAGER_CI_IMAGE}

echo "Create Kind cluster and load Kubeflow Trainer images"
${KIND} create cluster --image "${KIND_NODE_VERSION}"
cat <<EOF | ${KIND} create cluster --image "${KIND_NODE_VERSION}" --config=-
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added this config, so we can add more Kind workers in the future to decrease kube-scheduler time for placing Pods.
I think, that should speedup our tests once we run more TrainJobs at the same time.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you open a separate PR? Because this seems not to related to openapi-gen.
If we find any issue for Kind config adding, the dedicated PR allows us to easily revert and fix that.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, let me open it.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
EOF

${KIND} load docker-image ${CONTROLLER_MANAGER_CI_IMAGE}

echo "Deploy Kubeflow Trainer control plane"
Expand Down
53 changes: 25 additions & 28 deletions hack/python-sdk/gen-sdk.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,41 +21,30 @@ set -o nounset

# TODO (andreyvelich): Read this data from the global VERSION file.
SDK_VERSION="0.1.0"

SDK_OUTPUT_PATH="sdk"

SWAGGER_JAR_URL="https://repo1.maven.org/maven2/org/openapitools/openapi-generator-cli/4.3.1/openapi-generator-cli-4.3.1.jar"
SWAGGER_CODEGEN_JAR="hack/python-sdk/openapi-generator-cli.jar"
OPENAPI_GENERATOR_VERSION="v7.11.0"
TRAINER_ROOT="$(pwd)"
SWAGGER_CODEGEN_CONF="hack/python-sdk/swagger_config.json"
SWAGGER_CODEGEN_FILE="api/openapi-spec/swagger.json"

if [[ ! -f "$SWAGGER_CODEGEN_JAR" ]]; then
echo "Downloading the openapi-generator-cli JAR package to generate SDK"
wget -O "${SWAGGER_CODEGEN_JAR}" ${SWAGGER_JAR_URL}
fi

echo "Generating Python SDK for Kubeflow Trainer V2 ..."
java -jar "${SWAGGER_CODEGEN_JAR}" generate -i "${SWAGGER_CODEGEN_FILE}" -g python \
-o "${SDK_OUTPUT_PATH}" \
-c "${SWAGGER_CODEGEN_CONF}" \
# We need to add user to allow container override existing files.
docker run --user "$(id -u)":"$(id -g)" --rm \
-v "${TRAINER_ROOT}:/local" docker.io/openapitools/openapi-generator-cli:${OPENAPI_GENERATOR_VERSION} generate \
-g python \
-i "local/${SWAGGER_CODEGEN_FILE}" \
-c "local/${SWAGGER_CODEGEN_CONF}" \
-o "local/${SDK_OUTPUT_PATH}" \
-p=packageVersion="${SDK_VERSION}" \
--global-property apiTests=false,modelTests=false # TODO (andreyvelich): Discuss if we should use these test files.
--global-property models,modelTests=false,modelDocs=false,supportingFiles=__init__.py

echo "Removing unused files for the Python SDK"
git clean -f ${SDK_OUTPUT_PATH}/.openapi-generator
git clean -f ${SDK_OUTPUT_PATH}/.gitignore
git clean -f ${SDK_OUTPUT_PATH}/.gitlab-ci.yml
git clean -f ${SDK_OUTPUT_PATH}/git_push.sh
git clean -f ${SDK_OUTPUT_PATH}/.openapi-generator-ignore
git clean -f ${SDK_OUTPUT_PATH}/.travis.yml
git clean -f ${SDK_OUTPUT_PATH}/requirements.txt
git clean -f ${SDK_OUTPUT_PATH}/setup.cfg
git clean -f ${SDK_OUTPUT_PATH}/setup.py
git clean -f ${SDK_OUTPUT_PATH}/test-requirements.txt
git clean -f ${SDK_OUTPUT_PATH}/tox.ini
git clean -f ${SDK_OUTPUT_PATH}/.github
git clean -f ${SDK_OUTPUT_PATH}/test

# Revert the README since it is manually created.
git checkout ${SDK_OUTPUT_PATH}/README.md
# Revert manually created files.
git checkout ${SDK_OUTPUT_PATH}/kubeflow/trainer/__init__.py

# Manually modify the SDK version in the __init__.py file.
Expand All @@ -65,7 +54,15 @@ else
sed -i -e "s/__version__.*/__version__ = \"${SDK_VERSION}\"/" ${SDK_OUTPUT_PATH}/kubeflow/trainer/__init__.py
fi

# Kubeflow models must have Kubernetes models to perform serialization.
printf "\n# Import Kubernetes and JobSet models for the serialization. \n" >>${SDK_OUTPUT_PATH}/kubeflow/trainer/models/__init__.py
printf "from kubernetes.client import *\n" >>${SDK_OUTPUT_PATH}/kubeflow/trainer/models/__init__.py
printf "from jobset.models import *\n" >>${SDK_OUTPUT_PATH}/kubeflow/trainer/models/__init__.py
# The `model_config` property conflicts with Pydantic name.
# Therefore, we rename it to `model_config_crd`
TRAINJOB_SPEC_MODEL=${SDK_OUTPUT_PATH}/kubeflow/trainer/models/trainer_v1alpha1_train_job_spec.py
if [[ $(uname) == "Darwin" ]]; then
sed -i '' -e "s/model_config/model_config_crd/" ${TRAINJOB_SPEC_MODEL}
sed -i '' -e "s/model_config_crd = ConfigDict/model_config = ConfigDict/" ${TRAINJOB_SPEC_MODEL}
sed -i '' -e "s/kubeflow.trainer.models.trainer_v1alpha1_model_config_crd/kubeflow.trainer.models.trainer_v1alpha1_model_config/" ${TRAINJOB_SPEC_MODEL}
else
sed -i -e "s/model_config/model_config_crd/" ${TRAINJOB_SPEC_MODEL}
sed -i -e "s/model_config_crd = ConfigDict/model_config = ConfigDict/" ${TRAINJOB_SPEC_MODEL}
sed -i -e "s/kubeflow.trainer.models.trainer_v1alpha1_model_config_crd/kubeflow.trainer.models.trainer_v1alpha1_model_config/" ${TRAINJOB_SPEC_MODEL}
fi
6 changes: 1 addition & 5 deletions hack/python-sdk/swagger_config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
{
"packageName": "kubeflow.trainer",
"typeMappings": {
"K8sIoApiAutoscalingV2MetricSpec": "V2MetricSpec",
"K8sIoApimachineryPkgUtilIntstrIntOrString": "object",
"V1Time": "datetime"
}
"typeMappings": {}
}
Loading
Loading