Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inconsistent behaviour of helm install with a pre-install hook, helm version 3.17 #386

Open
mandymchu opened this issue Feb 10, 2025 · 1 comment

Comments

@mandymchu
Copy link

I have a helm chart which has a pre-install and pre-upgrade job to create a configmap. The template definition is something looks like

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-cm-sa
  namespace: {{ .Values.instanceNamespace }}
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-weight": "0"
  labels:
    component-id: {{ .Chart.Name }}
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: my-cm-role
  namespace: {{ .Values.instanceNamespace }}
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-weight": "0"
  labels:
    component-id: {{ .Chart.Name }}
rules: 
- apiGroups: [""]
  resources:
  - configmaps
  verbs:
  - get
  - create
  - update
  - patch
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: my-cm-rb
  namespace: {{ .Values.instanceNamespace }}
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-weight": "0"
  labels:
    component-id: {{ .Chart.Name }}
subjects:
  - kind: ServiceAccount
    name: my-cm-sa
    namespace: {{ .Values.instanceNamespace }}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: my-cm-role
---
apiVersion: batch/v1
kind: Job
metadata:
  name: my-cm-creation-job
  namespace: {{ .Values.instanceNamespace }}
  labels:
    swVersion: {{ .Values.releaseVersion }}
  annotations:
    "helm.sh/hook-delete-policy": before-hook-creation
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-weight": "1"
  labels:
    component-id: {{ .Chart.Name }}
spec:
  template:
    spec:
      containers:
      - name: dep
        image: '{{ .Values.imagePullPrefix }}/{{ .Values.migrationJobImgName }}@{{ .Values.migrationJobImgDigest }}'
        command:
          - bash
          - -c
          - |
            cd /opt/ansible/bin
            python3 create-my-cm.py --release={{ .Values.cpdPlatform.crVersion }} --namespace={{ .Values.instanceNamespace }}
      restartPolicy: Never
      serviceAccount: my-cm-sa
      imagePullSecrets:
      - name: {{ .Values.imagePullSecret }}

When I run helm install <my release> <my chart> to install my chart, it works fine if I run test using helm on mac. However, I hit inconsistent behaviour of the this command on linux. The helm version I used is v3.17.

  1. On linux environment, I hit an error complained about * roles.rbac.authorization.k8s.io "my-cm-role" not found and it cannot ignore it. This happened to me twice. see error message below
# helm install my-chart my-chart-5.2.0+20250205.211719.1.tgz -f ../top-override.yaml --debug
install.go:225: 2025-02-06 11:56:54.248796705 -0800 PST m=+0.082994631 [debug] Original chart version: ""
install.go:242: 2025-02-06 11:56:54.24901776 -0800 PST m=+0.083215656 [debug] CHART PATH: /root/helm/my-chart/my-chart-5.2.0+20250205.211719.1.tgz

client.go:486: 2025-02-06 11:56:54.802520336 -0800 PST m=+0.636718242 [debug] Starting delete for "my-cm-rb" RoleBinding
client.go:490: 2025-02-06 11:56:54.811039476 -0800 PST m=+0.645237392 [debug] Ignoring delete failure for "my-cm-rb" rbac.authorization.k8s.io/v1, Kind=RoleBinding: rolebindings.rbac.authorization.k8s.io "my-cm-rb" not found
wait.go:104: 2025-02-06 11:56:54.811112619 -0800 PST m=+0.645310535 [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:142: 2025-02-06 11:56:54.897877133 -0800 PST m=+0.732075049 [debug] creating 1 resource(s)
Error: INSTALLATION FAILED: failed pre-install: warning: Hook pre-install my-chart/templates/01-my-cm-job.yaml failed: 1 error occurred:
	* roles.rbac.authorization.k8s.io "my-cm-role" not found


helm.go:86: 2025-02-06 11:56:54.921988373 -0800 PST m=+0.756186289 [debug] failed pre-install: warning: Hook pre-install my-chart/templates/01-my-cm-job.yaml failed: 1 error occurred:
	* roles.rbac.authorization.k8s.io "my-cm-role" not found
  1. I used the exactly same code and same helm chart tgz file on another linux environment but this time helm installed the chart successfully
# helm install my-chart my-chart -f ../top-override.yaml --debug
install.go:225: 2025-02-06 12:49:47.85829741 -0800 PST m=+0.098835015 [debug] Original chart version: ""
install.go:242: 2025-02-06 12:49:47.858430573 -0800 PST m=+0.098968158 [debug] CHART PATH: /root/helm/my-chart/my-chart

client.go:486: 2025-02-06 12:49:48.286459443 -0800 PST m=+0.526997008 [debug] Starting delete for "my-cm-rb" RoleBinding
client.go:490: 2025-02-06 12:49:48.293423882 -0800 PST m=+0.533961447 [debug] Ignoring delete failure for "my-cm-rb" rbac.authorization.k8s.io/v1, Kind=RoleBinding: rolebindings.rbac.authorization.k8s.io "my-cm-rb" not found
wait.go:104: 2025-02-06 12:49:48.293510055 -0800 PST m=+0.534047620 [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:142: 2025-02-06 12:49:48.377448736 -0800 PST m=+0.617986321 [debug] creating 1 resource(s)
client.go:486: 2025-02-06 12:49:48.416271395 -0800 PST m=+0.656808960 [debug] Starting delete for "my-cm-role" Role
client.go:490: 2025-02-06 12:49:48.422204877 -0800 PST m=+0.662742462 [debug] Ignoring delete failure for "my-cm-role" rbac.authorization.k8s.io/v1, Kind=Role: roles.rbac.authorization.k8s.io "my-cm-role" not found
wait.go:104: 2025-02-06 12:49:48.422382402 -0800 PST m=+0.662919957 [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:142: 2025-02-06 12:49:48.506833396 -0800 PST m=+0.747370961 [debug] creating 1 resource(s)
client.go:486: 2025-02-06 12:49:48.538426129 -0800 PST m=+0.778963694 [debug] Starting delete for "my-cm-sa" ServiceAccount
client.go:490: 2025-02-06 12:49:48.545270465 -0800 PST m=+0.785808050 [debug] Ignoring delete failure for "my-cm-sa" /v1, Kind=ServiceAccount: serviceaccounts "my-cm-sa" not found
wait.go:104: 2025-02-06 12:49:48.545414469 -0800 PST m=+0.785952054 [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:142: 2025-02-06 12:49:48.712346587 -0800 PST m=+0.952884142 [debug] creating 1 resource(s)
client.go:486: 2025-02-06 12:49:48.749293818 -0800 PST m=+0.989831423 [debug] Starting delete for "my-cm-creation-job" Job
client.go:490: 2025-02-06 12:49:48.755915019 -0800 PST m=+0.996452584 [debug] Ignoring delete failure for "my-cm-creation-job" batch/v1, Kind=Job: jobs.batch "my-cm-creation-job" not found
wait.go:104: 2025-02-06 12:49:48.756068593 -0800 PST m=+0.996606148 [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:142: 2025-02-06 12:49:48.852400462 -0800 PST m=+1.092938047 [debug] creating 1 resource(s)
client.go:720: 2025-02-06 12:49:48.881483241 -0800 PST m=+1.122020826 [debug] Watching for changes to Job my-cm-creation-job with timeout of 5m0s
client.go:748: 2025-02-06 12:49:48.899831574 -0800 PST m=+1.140369179 [debug] Add/Modify event for my-cm-creation-job: ADDED
client.go:787: 2025-02-06 12:49:48.899968277 -0800 PST m=+1.140505862 [debug] my-cm-creation-job: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:748: 2025-02-06 12:49:48.925374541 -0800 PST m=+1.165912106 [debug] Add/Modify event for my-cm-creation-job: MODIFIED
client.go:787: 2025-02-06 12:49:48.925634287 -0800 PST m=+1.166171872 [debug] my-cm-creation-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:748: 2025-02-06 12:49:51.58346561 -0800 PST m=+3.824003215 [debug] Add/Modify event for my-cm-creation-job: MODIFIED
client.go:787: 2025-02-06 12:49:51.583661596 -0800 PST m=+3.824199191 [debug] my-cm-creation-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:748: 2025-02-06 12:49:52.585773134 -0800 PST m=+4.826310729 [debug] Add/Modify event for my-cm-creation-job: MODIFIED
client.go:787: 2025-02-06 12:49:52.585881297 -0800 PST m=+4.826418882 [debug] my-cm-creation-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:748: 2025-02-06 12:49:53.602352905 -0800 PST m=+5.842890490 [debug] Add/Modify event for my-cm-creation-job: MODIFIED
client.go:787: 2025-02-06 12:49:53.602461858 -0800 PST m=+5.842999453 [debug] my-cm-creation-job: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:748: 2025-02-06 12:49:53.637120811 -0800 PST m=+5.877658396 [debug] Add/Modify event for my-cm-creation-job: MODIFIED
client.go:142: 2025-02-06 12:49:53.637751667 -0800 PST m=+5.878289232 [debug] creating 7 resource(s)
NAME: my-chart
LAST DEPLOYED: Thu Feb  6 12:49:47 2025
NAMESPACE: instance-ns
STATUS: deployed
REVISION: 1
TEST SUITE: None

Not sure how to avoid this inconsistent helm install behaviour?

@mandymchu mandymchu changed the title Inconsistent behaviour of helm install with a pre-install hook Inconsistent behaviour of helm install with a pre-install hook, helm version 3.17 Feb 10, 2025
@gjenkins8
Copy link
Member

hi 👋 Is this an issue related to the Helm CLI? If so, can you please raise over on https://github.com/helm/helm as a Helm issue please

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants