Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gcp: add note on discard_unpacked_layers for GKE #2257

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

beraldoleal
Copy link
Member

@beraldoleal beraldoleal commented Jan 22, 2025

GKE uses discard_unpacked_layers=true for optimizations, starting with version 1.27. This can cause failed to extract layer sha256:xxx: not found errors with PeerPods. The note explains disabling this flag in containerd to avoid issues.

GKE uses discard_unpacked_layers=true for optimizations, starting with
version 1.27. This can cause "failed to extract layer sha256:xxx: not
found" errors with PeerPods. The note explains disabling this flag in
containerd to avoid issues.

Signed-off-by: Beraldo Leal <[email protected]>
@beraldoleal beraldoleal requested a review from a team as a code owner January 22, 2025 18:44
@beraldoleal
Copy link
Member Author

@wainersm, @stevenhorsman I did some tests with peerpods + GCP using devel branch and the only issue I found is related to #2254, also documented here.

My local tests shows the issue is caused by discard_unpacked_layers=true in containerd's configuration, which is set by default in GKE (and maybe kind) to save some space. According with the implementation:

DiscardUnpackedLayers is a boolean flag to specify whether to allow GC to
remove layers from the content store after successfully unpacking these
layers to the snapshotter.

Disabling this flag resolved the problem in my tests. So I think it might be worth adding a note to the documentation.

Copy link
Member

@stevenhorsman stevenhorsman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks okay to me. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants