Impossible to build the image with podman

[vitaliy-guliy]

Describe the bug

It is not possible to build the image with podman

Che version

next (development version)

Steps to reproduce

• Create a workspace with https://github.com/devfile/developer-images/tree/test-podman
• open a terminal, go to test directory
• try run podman with podman build .

Expected behavior

Podman should build the image

Runtime

OpenShift

Screenshots

[AObuchow]

Still looking into this. Some things I noticed so far:

• This problem doesn't happen with all images, for example I can successfully build a container based on alpine but not quay.io/devfile/universal-developer-image:latest
• On my own system, I noticed I was unable to build the provided Dockerfile as well:

$ cat Dockerfile
FROM quay.io/devfile/universal-developer-image:ubi8-latest

$ podman build -f Dockerfile
(...)
Copying blob cab8b52e242f done   | 
Copying blob 4f54068ca527 done   | 
Copying blob ef6ec184fd75 done   | 
Error: creating build container: writing blob: adding layer with blob "sha256:7b747ba046ed87bf97489e2a2cab5b558e261f45b4af36c3b24f58a9f67a66b0": processing tar file(potentially insufficient UIDs or GIDs available in user namespace (requested 301071:301071 for /usr/local/bin/LICENSE): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": lchown /usr/local/bin/LICENSE: invalid argument): exit status 1

However, I can get it to build with podman --storage-opt ignore_chown_errors=true build . Unfortunately, using --storage-opt ignore_chown_errors=true won't help us with this bug, as it causes all files in the container to be owned by the user who built the image. Additionally, you can't even use `ignore_chown_errors on Che as the vsf driver on doesn't support it.

[ibuziuk]

@benoitf any ideas why podman can not build the image? should we report the issue ?

[benoitf]

I think I saw in the past some errors when there were not enough ids available

but you can file a bug (and if possible try also using podman v5)

[AObuchow]

I've tested and confirmed that devfile/developer-images#167 fixes this issue.
Note: due to #22914 I had to apply the following workaround to get podman to build: rm -rf .local/share/ && rm /home/user/.stow_completed && /entrypoint.sh.

The default 5 Gi used for per-workspace storage is sadly not enough storage space to build the UDI image. I tried expanding my PVC to 16Gi and even that was not enough (though only 14.38Gi were used, I imagine you need >16Gi of storage to get the UDI built):

Copying blob 783eb3c8c36c done  
Error: creating build container: writing blob: adding layer with blob "sha256:2247289fd0abbf0ada70f319bb588b66e6f191339ccbb8a326bef48b58d0c146": creating read-only layer with ID "c749ef69551678c194ff16e4147bc1582d3eb7d25bd6c867db5a1d703b7e812d": no space left on device

That merits its own issue being opened, however. Also we might want to try out using the fuse-overlayfs storage driver to help here: https://che.eclipseprojects.io/2024/03/28/@david.kwon-fuse-storage-driver.html CC: @dkwon17

[dkwon17]

It's not possible today to use overlay (fuse-overlayfs) storage driver on the dogfooding cluster, unless we either:

• upgrade the cluster to OpenShift 4.15
• follow these steps to perform modifications to the cluster: https://eclipse.dev/che/docs/stable/administration-guide/configuring-fuse/

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.