Sometimes Kubelet keeps retrying volume unmount causing excessive logs

[mugdha-adhav]

Background

Sometimes kubelet gets stuck in a loop trying to unmount a volume by calling NodeUnpublishVolume(). This causes excessive error logs to be generated by kubelet.

Suggested approaches

1. Improve/fix error handling in NodeUnpublishVolume function.
2. Add synchronicity in unmount calls using locks, mutexes or by tracking inFlight requests.

Additional information

The error message looks something like -

kubelet: E0108 07:04:50.385776 4499 reconciler_common.go:169] "operationExecutor.UnmountVolume failed (controllerAttachDetachEnabled true) for volume \"xyz\" (UniqueName: \"kubernetes.io/csi/3a735360-c5b8-4eb5-8f4d-c8191e76bb9b-xyz\") pod \"3a735360-c5b8-4eb5-8f4d-c8191e76bb9b\" (UID: \"3a735360-c5b8-4eb5-8f4d-c8191e76bb9b\") : UnmountVolume.NewUnmounter failed for volume \"xyz\" (UniqueName: \"kubernetes.io/csi/3a735360-c5b8-4eb5-8f4d-c8191e76bb9b-xyz\") pod \"3a735360-c5b8-4eb5-8f4d-c8191e76bb9b\" (UID: \"3a735360-c5b8-4eb5-8f4d-c8191e76bb9b\") : kubernetes.io/csi: unmounter failed to load volume data file [/var/lib/kubelet/pods/3a735360-c5b8-4eb5-8f4d-c8191e76bb9b/volumes/kubernetes.io<del>csi/xyz/mount]: kubernetes.io/csi: failed to open volume data file [/var/lib/kubelet/pods/3a735360-c5b8-4eb5-8f4d-c8191e76bb9b/volumes/kubernetes.io</del>csi/xyz/vol_data.json]: open /var/lib/kubelet/pods/3a735360-c5b8-4eb5-8f4d-c8191e76bb9b/volumes/kubernetes.io~csi/xyz/vol_data.json: no such file or directory"

Related kubelet issue: kubernetes/kubernetes#116847 (comment)