Allow prioritize user-addons / helm-releases

[kron4eg]

Description of the feature you would like to add / User story

Currently we have useraddons deployed before the helm releases. But in case when CNI is set to be external and the actual CNI driver is deployed via the helmRelease, nodes will not be Ready until helm release happened.

If we simply change the order, the vise-versa problem will be true in case when CNI driver is deployed via the addon.

Solution details

Unify the logic to deploy useraddons and helm releases. Add weight fields to both Addon and HelmRelease types. Consider this field as a priority when deploying.

Alternative approaches

Add new type in KubeOneCluster workloads that will unify under one umbrella all existing and future types of deploys (addons/helmReleases). Deprecate addons.addons and helmReleases.

examples:

addons:
  enable: true
  addons: ## this will be deprecated, yet still working for some time
  - name: default-storage-class
    weight: 100

helmReleases:  ## this will be deprecated, yet still working for some time
  - chart: kured
    repoURL: https://kubereboot.github.io/charts
    namespace: kube-system
    version: 5.4.5
    weight: 500

## this will make a unified list naturally defining the order of deployment
workloads:
  - addon:
      name: backups-restic
  - addon:
      name: unattended-upgrades
  - helmChart:
      chart: kured
      repoURL: https://kubereboot.github.io/charts
      namespace: kube-system
      version: 5.4.5
  - addon:
      name: default-storage-class

[kron4eg]

Closed in #3248

[xmudrii]

I don't think that we ever fixed this issue because looking at

kubeone/pkg/tasks/tasks.go

Lines 279 to 293 in 8b8d794

	{
	Fn: addons.Ensure,
	Operation: "applying addons",
	Description: "ensure embedded addons",
	},
	{
	Fn: addons.EnsureUserAddons,
	Operation: "applying addons",
	Description: "ensure custom addons",
	Predicate: func(s *state.State) bool { return s.Cluster.Addons != nil },
	},
	{
	Fn: localhelm.Deploy,
	Operation: "releasing core helm charts",
	},

it still looks like we always first deploy addons, so you can't deploy any Helm chart before the addons.

If I'm mistaken, @kron4eg can you please provide an example of how this works in the new API version?
/reopen

[kubermatic-bot]

@xmudrii: Reopened this issue.

In response to this:

I don't think that we ever fixed this issue because looking at

kubeone/pkg/tasks/tasks.go

Lines 279 to 293 in 8b8d794

	{
	Fn: addons.Ensure,
	Operation: "applying addons",
	Description: "ensure embedded addons",
	},
	{
	Fn: addons.EnsureUserAddons,
	Operation: "applying addons",
	Description: "ensure custom addons",
	Predicate: func(s *state.State) bool { return s.Cluster.Addons != nil },
	},
	{
	Fn: localhelm.Deploy,
	Operation: "releasing core helm charts",
	},

, it still looks like we always first deploy addons, so you can't deploy any Helm chart before the addons.

If I'm mistaken, @kron4eg can you please provide an example of how this works in the new API version?
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.