Publish helm charts to artifact registry

[ahg-g]

We can do this to the staging artifact registry we have for jobset similar to Kueue:

See the makefile rule here: https://github.com/kubernetes-sigs/kueue/blob/main/Makefile#L215; and then we add it to the cloudbuild.yaml file like https://github.com/kubernetes-sigs/kueue/blob/main/cloudbuild.yaml#L13

[tenzen-y]

Just FYI: If we do not promote the Chart similar to container images, the Chart will be removed in 90 days. Additionally, the Chart promotion way is different from container image promotion (see: more detail kubernetes-sigs/kueue#4143)

[ahg-g]

Can we follow the same promotion process as Kueue and publish to https://github.com/kubernetes/k8s.io/tree/main/registry.k8s.io/images/charts ?

btw, why is that path not qualified with a subproject name?

[tenzen-y]

Can we follow the same promotion process as Kueue and publish to https://github.com/kubernetes/k8s.io/tree/main/registry.k8s.io/images/charts ?

I think so too.

btw, why is that path not qualified with a subproject name?

As we discussed in https://docs.google.com/document/d/1xa2me_O0N5sHMD_np9ztr2O9GL3uX1yKBgDcApDqV_8/edit?usp=sharing&resourcekey=0-JHI0r4CeitKdIEJII7c8eg, we evaluate some solutions with sig k8s-infra.
In conclusion, we follow the recommendation from sig- k8s-infra folks to provide full compatibility of Helm specifications (mostly image path specification like charts/$(PROJECT) instead of $(PROJECT)/charts).

[kannon92]

/assign

i started this work #792

[ChenYi015]

What about using the netlify doc website to host the Helm chart repo index file? We can add a GitHub action workflow to do that, once a new release is published, a helm-chart-releaser workflow will be triggered to update the Helm repo index file which will be placed at site/content/en/charts/index.yaml, then users can install the Helm chart as follows:

# Add Helm repo
helm repo add jobset https://jobset.sigs.k8s.io/charts

# Update Helm repo index
helm repo update jobset

# Search Helm repo for stable release versions of jobset
helm search repo jobset
# NAME                    CHART VERSION   APP VERSION             DESCRIPTION
# jobset/jobset           0.8.0                                   A Helm chart for deploying jobset on Kubernetes.

# Install the latest released version of jobset
helm install jobset jobset/jobset \
    --namespace jobset-system \
    --create-namespace

# Or, install jobset with the specified version
helm install jobset jobset/jobset \
    --version 0.8.0 \
    --namespace jobset-system \
    --create-namespace

[andreyvelich]

@ChenYi015 Does it require to host netlify doc website as part of every repo where we want to publish Helm Charts ?
If possible, I would like us to design solution that can work even if we don't host website in the repo.
What do we do for Spark Operator since docs are located in the kubeflow.org ?
https://github.com/kubeflow/spark-operator/tree/gh-pages

[ChenYi015]

Does it require to host netlify doc website as part of every repo where we want to publish Helm Charts ?

We can also use GitHub pages to host the Helm repo index file. Spark operator publish helm charts in this way, see https://github.com/kubeflow/spark-operator/blob/gh-pages/index.yaml. Another example is prometheus-operator, the Prometheus community creates a dedicated repo prometheus-community/helm-charts and use GitHub pages to host the Helm chart index file, see https://github.com/prometheus-community/helm-charts/blob/gh-pages/index.yaml.

[andreyvelich]

I actually, like how Prometheus handle it.
Do you know if Argo does the same way https://github.com/argoproj/argo-helm/tree/gh-pages ?
Or they also publish them to ghcr.io/argoproj/argo-helm ?

[tenzen-y]

ArtifactRegistry hosting has different objectives. The goal is to support Helm OCI features like helm install jobset oci://registry.k8s.io/charts/jobset --version 0.8.0. This is just one command, there is no needed helm repo add and helm repo update.

This way provides more CI/CD automated deploy mechanism compatibility. Actually, the requested users use FluxCD.

[tenzen-y]

If JobSet does not aim to support the Helm OCI feature, we can take any approach, including GH pages, although we need to build another publishing mechanism for GH pages, which means we need to maintain GH actions.

[tenzen-y]

Note that I do not have any preference here. All things are just comments.

[kannon92]

I want consistency with Kueue, JobSet and LWS. So I lean with going with @tenzen-y 's suggestion.

[ChenYi015]

Do you know if Argo does the same way https://github.com/argoproj/argo-helm/tree/gh-pages ?
Or they also publish them to ghcr.io/argoproj/argo-helm ?

Yes, it would be better if we can also publish Helm charts in both two ways, just like Prometheus, Argo and Spark Operator has did.

Just FYI, we have already setup an workflow in Spark operator to publish Helm charts both to the release artifacts (use GH pages to host index file) and GHCR, see https://github.com/kubeflow/spark-operator/blob/79264a4ac559cd6373b43bb54365d566b2537acb/.github/workflows/release-helm-charts.yaml#L50-L84.

[kannon92]

I was going to follow @tenzen-y work with Kueue for this release. We can iterate on this but i'd like to get a release out and this seems to be the solution we have at the moment.

[ahg-g]

where do we have the charts published now? I don't see them under https://pantheon.corp.google.com/artifacts/docker/k8s-staging-images/us-central1/jobset yet, I thought they will be created on each commit just like the controller image

[kannon92]

I can't view that link.

[kannon92]

https://console.cloud.google.com/artifacts/docker/k8s-staging-images/us-central1/charts/jobset?invt=Abqotg&inv=1

I found them here.

[ahg-g]

ah, cool, I was looking at https://console.cloud.google.com/artifacts/docker/k8s-staging-images/us-central1/jobset