Failure to Extract Layers in Containerd

[beraldoleal]

Describe the bug

In a GKE cluster with containerd and Ubuntu (1.7.24-0ubuntu0~22.04.1), containerd fails to extract image layers during container creation.

This occurs even when image is successfully pulled and present in the content store. Running ctr content fetch ... manually on node fixes the issue, but this requires manual intervention.

Container creation could seamlessly pull and unpack all image layers without requiring manual fetch commands.

** Note: This might not be specific to GCP and could be a combination of containerd and Nydus version?

How to reproduce

1. Create a GKE cluster using the following command:

gcloud container clusters create my-cluster \
  --zone ${GCP_REGION}-a \
  --machine-type "e2-standard-4" \
  --image-type UBUNTU_CONTAINERD \
  --num-nodes 1

2. Deploy a pod using an image from a public registry (e.g., quay.io/prometheus/busybox) and kata-remote
3. Observe the container creation logs for failures during image unpacking.

CoCo version information

caa HEAD, containerd 1.7.24

What TEE are you seeing the problem on

None

Failing command and relevant log output

failed to extract layer sha256:<layer-digest>: failed to get reader from content store: content digest sha256:<digest>: not found