Only one node picked up by cluster-init which also fails with Forbidden Error

[mmnelemane]

My cmk-cluster-init-pod.yaml looks as below. I am trying to initialize a cluster with 3 worker nodes. However, only one of the node (strangely only "worker-1" which is the second in the list) has been installed with cmk and related binaries. I could find the logs for the cluster-init-pod from this node which shows a Forbidden error (403) as shown below

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: cmk-cluster-init-pod
  name: cmk-cluster-init-pod
  namespace: cmk-namespace
spec:
  serviceAccountName: cmk-serviceaccount
  containers:
  - args:
      # Change this value to pass different options to cluster-init.
      - "/cmk/cmk.py cluster-init --host-list=worker-0,worker-1,worker-2"
    command:
    - "/bin/bash"
    - "-c"
    image: mjace/cmk:v1.3.1
    name: cmk-cluster-init-pod
    securityContext:
      privileged: false
    ports:
    - containerPort: 8080
  restartPolicy: Never

cluster-init issues the following error on worker-1 node:

2019-12-30T16:45:13.333207607+00:00 stderr F INFO:root:Used ServiceAccount: cmk-serviceaccount
2019-12-30T16:45:13.333207607+00:00 stderr F INFO:root:Creating cmk pod for ['init', 'install', 'discover'] commands ...
2019-12-30T16:45:13.449979685+00:00 stderr F ERROR:root:Exception when creating pod for ['init', 'install', 'discover'] command(s): (403)
2019-12-30T16:45:13.449979685+00:00 stderr F Reason: Forbidden
2019-12-30T16:45:13.449979685+00:00 stderr F HTTP response headers: HTTPHeaderDict({'Content-Length': '301', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 30 Dec 2019 16:45:13 GMT'})
2019-12-30T16:45:13.449979685+00:00 stderr F HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User "system:serviceaccount:cmk-namespace:cmk-serviceaccount" cannot create resource "pods" in API group "" in the namespace "default"","reason":"Forbidden","details":{"kind":"pods"},"code":403}
2019-12-30T16:45:13.449979685+00:00 stderr F
2019-12-30T16:45:13.449979685+00:00 stderr F
2019-12-30T16:45:13.449979685+00:00 stderr F ERROR:root:Aborting cluster-init ...

I am currently using k8s v1.5.2 and cmk images from v1.3.1.

[przemeklal]

If you're using cluster-init to install CMK on your cluster in a namespace please don't forget to add --namespace=cmk-namespace argument. This way cluster-init will try to create CMK pods in the cmk-namespace instead of using default where it doesn't have permissions to do so.

Beyond that the scheduling is not an issue here - cluster-init always runs on a single node only and after that, it schedules the CMK reconcile and nodereport daemonset on the nodes you pass in the --host-list - this is exactly the step that fails in cluster-init due to RBAC issues, so I wouldn't worry about that now.

CMK v1.4.0 is already released and has a lot of fixes so highly recommend using that. If possible it would also make sense to use some newer version of Kubernetes - which has support for more interesting features like custom resources or support for dynamic admission controller (cmk-webhook).