microk8s is not starting in Ubuntu 20.04

[eldhogeorge86]

I am trying to setup microk8s in a VM. I ran the below command to setup.
sudo snap install microk8s --classic

microk8s status is showing "microk8s is not running. Use microk8s inspect for a deeper inspection."

I saw the below error in snap.microk8s.daemon-apiserver-kicker.

Feb 11 17:55:46 INBLRH14371LNPR systemd[1]: Started Service for snap application microk8s.daemon-apiserver-kicker.
Feb 11 17:55:51 INBLRH14371LNPR microk8s.daemon-apiserver-kicker[6001]: Setting up the CNI
Feb 11 17:56:26 INBLRH14371LNPR microk8s.daemon-apiserver-kicker[6382]: [unable to recognize "/var/snap/microk8s/2948/args/cni-network/cni.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1", unable to recognize "/var/snap/microk8s/2948/args/cni-network/cni.yaml": no matches for kind "ClusterRole" in version "rbac.authorization.k8s.io/v1", unable to recognize "/var/snap/microk8s/2948/args/cni-network/cni.yaml": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1", unable to recognize "/var/snap/microk8s/2948/args/cni-network/cni.yaml": no matches for kind "DaemonSet" in version "apps/v1", unable to recognize "/var/snap/microk8s/2948/args/cni-network/cni.yaml": no matches for kind "Deployment" in version "apps/v1", unable to recognize "/var/snap/microk8s/2948/args/cni-network/cni.yaml": no matches for kind "PodDisruptionBudget" in version "policy/v1beta1"]
Feb 11 17:56:26 INBLRH14371LNPR microk8s.daemon-apiserver-kicker[6382]: Error from server (Forbidden): error when creating "/var/snap/microk8s/2948/args/cni-network/cni.yaml": configmaps "calico-config" is forbidden: not yet ready to handle request
Feb 11 17:56:26 INBLRH14371LNPR microk8s.daemon-apiserver-kicker[6382]: Error from server (Forbidden): error when creating "/var/snap/microk8s/2948/args/cni-network/cni.yaml": serviceaccounts "calico-node" is forbidden: not yet ready to handle request
Feb 11 17:56:26 INBLRH14371LNPR microk8s.daemon-apiserver-kicker[6382]: error when creating "/var/snap/microk8s/2948/args/cni-network/cni.yaml": Post "https://127.0.0.1:16443/api/v1/namespaces/kube-system/serviceaccounts?fieldManager=kubectl-client-side-apply&timeout=2m0s": unexpected EOF

Also this error in snap.microk8s.daemon-containerd

Feb 11 17:55:46 INBLRH14371LNPR microk8s.daemon-containerd[5716]: time="2022-02-11T17:55:46.029002236+05:30" level=info msg="Get image filesystem path \"/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs\""
Feb 11 17:55:46 INBLRH14371LNPR microk8s.daemon-containerd[5716]: time="2022-02-11T17:55:46.029380310+05:30" level=error msg="failed to load cni during init, please check CRI plugin status before setting up network for pods" error="cni config load failed: no network config found in /var/snap/microk8s/2948/args/cni-network: cni plugin not initialized: failed to load cni config"
Feb 11 17:55:46 INBLRH14371LNPR microk8s.daemon-containerd[5716]: time="2022-02-11T17:55:46.029417048+05:30" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1

The cni.yaml is present in this directory "/var/snap/microk8s/2948/args/cni-network"

Please run microk8s inspect and attach the generated tarball to this issue.

We appreciate your feedback. Thank you for using microk8s.
inspection-report-20220211_185255.tar.gz

[balchua]

Is it possible to change your hostname to all small letters? Kubernetes normalizes it to all small. Once you do that, you may want to remove microk8s sudo snap remove microk8s --purge.

Then install it again.

[eldhogeorge86]

I have tried changing the hostname and reinstalled the microk8s. But still getting the same error.

inspection-report-20220215_110435.tar.gz

[balchua]

Thanks for updating. Quick question, did you remove some of the logs produced by the inspect tarball, example the dqlite or containerd etc..? Just checking.

[eldhogeorge86]

No, nothing was removed from the tarball. I just created a new tarball and uploaded again, incase something is missing.

microk8s inspect

Inspecting Certificates
Inspecting services
  Service snap.microk8s.daemon-cluster-agent is running
  Service snap.microk8s.daemon-containerd is running
  Service snap.microk8s.daemon-k8s-dqlite is running
  Service snap.microk8s.daemon-kubelite is running
  Service snap.microk8s.daemon-apiserver-kicker is running
  Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
  Copy processes list to the final report tarball
  Copy snap list to the final report tarball
  Copy VM name (or none) to the final report tarball
  Copy disk usage information to the final report tarball
  Copy memory usage information to the final report tarball
  Copy server uptime to the final report tarball
  Copy current linux distribution to the final report tarball
  Copy openSSL information to the final report tarball
  Copy network configuration to the final report tarball
Inspecting kubernetes cluster
  Inspect kubernetes cluster
Inspecting juju
  Inspect Juju
Inspecting kubeflow
  Inspect Kubeflow
Inspecting dqlite
  Inspect dqlite

Building the report tarball
  Report tarball is at /var/snap/microk8s/2948/inspection-report-20220216_023421.tar.gz

inspection-report-20220216_023421.tar.gz

[balchua]

Very strange. The only journal log available is kubelite. It doesn't have containerd, apiservice-kicker and dqlite. There is enuf space in the file system tho.

[eldhogeorge86]

I am able to see journal logs of containerd, apiservice-kicker and dqlite in the tarball.
See screenshot:

[ktsakalozos]

I think @balchua means that the time in the logs of k8s-dqlite is an hour behind the one on the kubelite. This makes me wonder if there is some time zone configuration we should tune.

Something else I have not seen before is the volume mounted in /var/.

[byroncollins]

I had the same issue
Removing and downgrading to 1.22/stable got me back up and running again

sudo snap remove microk8s --purge sudo snap install microk8s --classic --channel=1.22/stable

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ming-afk]

I had the same issue Removing and downgrading to 1.22/stable got me back up and running again

sudo snap remove microk8s --purge sudo snap install microk8s --classic --channel=1.22/stable

This worked for me. I then did a refresh to the version I want:
sudo snap refresh microk8s --classic --channel=1.28

[Bruno-Dompro]

Hi,

Just new in This world...
I installed new Ubuntu server (22.0.4 LTS) , snapd and microk8s.
But I'm stuck on "microk8s is not running. Use microk8s inspect for a deeper inspection."
I don't know what to do with the tarball generated.

Someone to have a look and tell me what is going wrong ?

Some Complements:

~$ snap list
Name Version Rev Tracking Publisher Notes
core20 20240111 2182 latest/stable canonical✓ base
lxd 5.0.3-babaaf8 27948 5.0/stable/… canonical✓ -
microk8s v1.29.2 6641 1.29/stable canonical✓ classic
snapd 2.61.2 21184 latest/stable canonical✓ snapd
~$

inspection-report-20240409_115121.tar.gz