Unexpected Microk8s 1.28.14 restart with cert update logs even though i have no-cert-reissue in-place

[chirag-launchnodes]

Unexpected MicroK8s Restart Despite Update and Certificate Lock Controls

We are using static IP for this single node microk8s node.
We recently experienced an unexpected MicroK8s restart that occurred without any clear trigger. We have observed two known patterns where MicroK8s restarts:
When Snap updates MicroK8s:
To resolve this, we have manually disabled automatic updates using the following command:

snap refresh --hold microk8s

When Snap attempts to renew MicroK8s certificates:
We have addressed this by preventing certificate renewals using the lock file as per abvoe, with the following command:

touch /var/snap/microk8s/current/var/lock/no-cert-reissue

However, despite these measures, we still experienced a restart today. Upon reviewing the logs, it appears that Snap attempted to renew the certificates, but the lock file successfully prevented this. Despite this, the restart still occurred. Below are relevant syslogs at the time of the incident

snapd[1595379]: storehelpers.go:923: cannot refresh: snap has no updates available: "certbot", "lxd", "microk8s", "snapd"
solo-staking-dev01 systemd[1]: Reloading.
systemd[1]: Configuration file /run/systemd/system/netplan-ovs-cleanup.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.|
systemd[1]: Configuration file /etc/systemd/system/snap.microk8s.daemon-kubelite.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
systemd[1]: Starting Daily apt download activities...
systemd[1]: Mounting Mount unit for core20, revision 2434...
kernel: [7741074.837352] loop1: detected capacity change from 0 to 130448
systemd[1]: Mounted Mount unit for core20, revision 2434.
Configuration file /etc/systemd/system/snap.microk8s.daemon-kubelite.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
microk8s.daemon-apiserver-kicker[966191]: /snap/microk8s/7231/actions/common/utils.sh: line 582: /snap/microk8s/7231/bin/sed: No such file or directory
microk8s.daemon-apiserver-kicker[966190]: /snap/microk8s/7231/actions/common/utils.sh: line 582: /snap/microk8s/7231/bin/hostname: No such file or directory
microk8s.daemon-apiserver-kicker[966194]: /snap/microk8s/7231/actions/common/utils.sh: line 586: /snap/microk8s/7231/bin/grep: No such file or directory
microk8s.daemon-apiserver-kicker[966196]: /snap/microk8s/7231/actions/common/utils.sh: line 586: /snap/microk8s/7231/usr/bin/cut: No such file or directory
microk8s.daemon-apiserver-kicker[966197]: /snap/microk8s/7231/actions/common/utils.sh: line 586: /snap/microk8s/7231/usr/bin/head: No such file or directory
microk8s.daemon-apiserver-kicker[966195]: /snap/microk8s/7231/actions/common/utils.sh: line 586: /snap/microk8s/7231/usr/bin/gawk: No such file or directory
microk8s.daemon-apiserver-kicker[966200]: /snap/microk8s/7231/actions/common/utils.sh: line 586: /snap/microk8s/7231/bin/grep: No such file or directory
microk8s.daemon-apiserver-kicker[966201]: /snap/microk8s/7231/actions/common/utils.sh: line 586: /snap/microk8s/7231/usr/bin/gawk: No such file or directory
microk8s.daemon-apiserver-kicker[966202]: /snap/microk8s/7231/actions/common/utils.sh: line 586: /snap/microk8s/7231/usr/bin/cut: No such file or directory
microk8s.daemon-apiserver-kicker[966203]: /snap/microk8s/7231/actions/common/utils.sh: line 586: /snap/microk8s/7231/usr/bin/head: No such file or directory
microk8s.daemon-apiserver-kicker[966206]: /snap/microk8s/7231/actions/common/utils.sh: line 586: /snap/microk8s/7231/bin/grep: No such file or directory

microk8s.daemon-apiserver-kicker[966207]: /snap/microk8s/7231/actions/common/utils.sh: line 586: /snap/microk8s/7231/usr/bin/gawk: No such file or directory
microk8s.daemon-apiserver-kicker[966208]: /snap/microk8s/7231/actions/common/utils.sh: line 586: /snap/microk8s/7231/usr/bin/cut: No such file or directory
microk8s.daemon-apiserver-kicker[966209]: /snap/microk8s/7231/actions/common/utils.sh: line 586: /snap/microk8s/7231/usr/bin/head: No such file or directory
microk8s.daemon-apiserver-kicker[966212]: /snap/microk8s/7231/actions/common/utils.sh: line 586: /snap/microk8s/7231/bin/grep: No such file or directory
microk8s.daemon-apiserver-kicker[966213]: /snap/microk8s/7231/actions/common/utils.sh: line 586: /snap/microk8s/7231/usr/bin/gawk: No such file or directory
microk8s.daemon-apiserver-kicker[966214]: /snap/microk8s/7231/actions/common/utils.sh: line 586: /snap/microk8s/7231/usr/bin/cut: No such file or directory
microk8s.daemon-apiserver-kicker[966215]: /snap/microk8s/7231/actions/common/utils.sh: line 586: /snap/microk8s/7231/usr/bin/head: No such file or directory
microk8s.daemon-apiserver-kicker[966216]: /snap/microk8s/7231/actions/common/utils.sh: line 800: /snap/microk8s/7231/bin/cp: No such file or directory
microk8s.daemon-apiserver-kicker[966217]: /snap/microk8s/7231/actions/common/utils.sh: line 815: /snap/microk8s/7231/bin/sed: No such file or directory
microk8s.daemon-apiserver-kicker[966218]: /snap/microk8s/7231/actions/common/utils.sh: line 788: /snap/microk8s/7231/bin/cp: No such file or directory
microk8s.daemon-apiserver-kicker[966220]: /snap/microk8s/7231/actions/common/utils.sh: line 603: /snap/microk8s/7231/usr/bin/openssl: No such file or directory
microk8s.daemon-apiserver-kicker[966221]: /snap/microk8s/7231/actions/common/utils.sh: line 604: /snap/microk8s/7231/usr/bin/openssl: No such file or directory
microk8s.daemon-apiserver-kicker[966224]: /snap/microk8s/7231/actions/common/utils.sh: line 608: /snap/microk8s/7231/usr/bin/openssl: No such file or directory
microk8s.daemon-apiserver-kicker[966225]: /snap/microk8s/7231/actions/common/utils.sh: line 608: /snap/microk8s/7231/bin/sed: No such file or directory
microk8s.daemon-apiserver-kicker[966226]: /snap/microk8s/7231/actions/common/utils.sh: line 609: /snap/microk8s/7231/usr/bin/openssl: No such file or directory
microk8s.daemon-apiserver-kicker[1597100]: cert change detected. Restarting the cluster-agent
systemd[1]: Stopping Service for snap application microk8s.daemon-cluster-agent...
systemd[1]: snap.microk8s.daemon-cluster-agent.service: Killing process 1597325 (cluster-agent) with signal SIGKILL.

[ktsakalozos]

Hi @chirag-launchnodes, thank you for reporting this. Since you have hold the refreshes there should not have been a restart. The error you pasted here shows that the new snap revision 7231 indeed did not land on you machine but the apiservice-kicker thinks it did so it tries to find binaries in the wrong path. I have opened an issue with the snap teams to try to understand what happened. You can follow up in https://forum.snapcraft.io/t/snap-hold-refresh-and-snap-data/43877