podman: out of box podman run failures on both 24.10.0-rc5 and 23.05.5

[adelton]

Maintainer: @oskarirauta
Environment: (put here arch, model, OpenWrt version)

x86_64 VM, OpenWrt 24.10.0-rc5 (r28304-6dacba30a7) or OpenWrt 23.05.5 (r24106-10cc5fcd00)

Description:

On freshly installed / reset OpenWrt 24.100-rc5 or 23.05.5 with podman installed, basic podman run fails:

OpenWrt 24.10.0-rc5

root@OpenWrt:~# podman run --rm -ti docker.io/library/busybox date
Trying to pull docker.io/library/busybox:latest...
Getting image source signatures
Copying blob 9c0abc9c5bd3 done   | 
Copying config af47096251 done   | 
Writing manifest to image destination
Error: netavark: iptables: No such file or directory (os error 2)

# podman version
Client:       Podman Engine
Version:      5.2.2
API Version:  5.2.2
Go Version:   go1.23.4
Built:        Fri Jan 10 13:57:59 2025
OS/Arch:      linux/amd64

OpenWrt 23.05.5

root@OpenWrt:~# podman run --rm -ti docker.io/library/busybox date
ERRO[0000] Rolling back transaction to validate database: sql: transaction has already been committed or rolled back 
Error: database static dir "/var/lib/containers/storage/libpod" does not match our static dir "/tmp/lib/containers/storage/libpod": database configuration mismatch

root@OpenWrt:~# podman version
ERRO[0000] Rolling back transaction to validate database: sql: transaction has already been committed or rolled back 
Error: database static dir "/var/lib/containers/storage/libpod" does not match our static dir "/tmp/lib/containers/storage/libpod": database configuration mismatch

I can likely debug and fix either of these issues but before doing that I'd like to calibrate expectations about the behaviour in the default configuration. Are these unheard of / unexpected, or expected after-fresh-install results?

[adelton]

As for the 23.05.5 which has

# podman version
Client:       Podman Engine
Version:      4.8.0
API Version:  4.8.0
Go Version:   go1.21.13
Built:        Wed Jan  8 18:59:28 2025
OS/Arch:      linux/amd64

The workaround seems to be to

root@OpenWrt:~# rm -f /tmp/lib/containers/storage/db.sql

The file gets recreated upon the next podman operation with updated paths, so after that the podman run command shown above works.

This seems to be an instance of containers/podman#20872 which got fixed in 4.8 branch with containers/podman#20888 and the fix released in https://github.com/containers/podman/releases/tag/v4.8.1. I wonder if OpenWrt 23 podman package could / should get upgraded to the latest 4.8 release, which at this time seems to be https://github.com/containers/podman/releases/tag/v4.8.3, a security release.

[adelton]

On 24.10.0-rc5, the

root@OpenWrt:~# podman run --log-level=debug --rm docker.io/library/busybox date 2>&1 | grep firewall
[INFO  netavark::firewall] Using iptables firewall driver

suggests that netavark decided to use iptables driver even if iptables is not installed, and

root@OpenWrt:~# grep firewall /etc/containers/containers.conf 
#firewall_driver = "none"

shows that it is not explicitly configured. The https://github.com/containers/common/blob/main/docs/containers.conf.5.md says

The firewall driver to be used by netavark. The default is empty which means netavark will pick one accordingly.

so the fact that podman 5.2 on OpenWrt 24 picks wrong is worrying.

Luckily the solution to move us forward on OpenWrt 24 seems quite simple -- just force the use of nftables:

root@OpenWrt:~# podman run --rm -ti docker.io/library/busybox date
Error: netavark: iptables: No such file or directory (os error 2)
root@OpenWrt:~# sed -i '/^#firewall_driver/afirewall_driver = "nftables"' /etc/containers/containers.conf
root@OpenWrt:~# podman run --rm -ti docker.io/library/busybox date
Error: netavark: IO error: Error while applying dns entries: IO error: aardvark-dns failed to start: Error from child process
Error starting server failed to bind udp listener on 10.88.0.1:53: IO error: Address in use (os error 98)

---

For comparison, the default configuration on 23.05.5 is the same

root@OpenWrt:~# grep firewall /etc/containers/containers.conf
#firewall_driver = "none"

but netavark says it was forced to use none:

root@OpenWrt:~# podman run --log-level=debug --rm docker.io/library/busybox date 2>&1 | grep firewall
[DEBUG netavark::firewall] Forcibly using firewall driver none
[INFO  netavark::firewall] Not using firewall

I wouldn't call it "forced" but maybe back then the default were different.

[adelton]

So we are able to solve the firewall_driver problem on OpenWrt 24 but we have a new one -- the UDP port 53.

Clearly podman (or rather, aardvark-dns) clashes with dnsmasq:

root@OpenWrt:~# podman run --rm -ti docker.io/library/busybox date
Error: netavark: IO error: Error while applying dns entries: IO error: aardvark-dns failed to start: Error from child process
Error starting server failed to bind udp listener on 10.88.0.1:53: IO error: Address in use (os error 98)
root@OpenWrt:~# /etc/init.d/dnsmasq stop
root@OpenWrt:~# podman run --rm -ti docker.io/library/busybox date
Tue Jan 14 15:19:47 UTC 2025

But people will often want to have both dnsmasq and podman, so the wholesale stop (and disable) of dnsmasq is likely not the ideal solution.

[adelton]

Checking with ip a and netstat -ulnp we can see that a podman0 device was created with IP address 10.88.0.1, and dnsmasq listens on that IP address, among others.

The solution seems to be to mark that interface as excluded for dnsmasq:

root@OpenWrt:~# uci add_list dhcp.@dnsmasq[0].notinterface=podman0
root@OpenWrt:~# uci commit
root@OpenWrt:~# /etc/init.d/dnsmasq restart
[...]
root@OpenWrt:~# podman run --rm docker.io/library/busybox date
Tue Jan 14 15:35:52 UTC 2025

---

This configuration seems to work but is not beautiful because it cannot be configured (nor seen) on the LuCI UI -- since the podman0 is not configured as a permanent interface, LuCI does not know about it. The solution might be to create the device and interface:

root@OpenWrt:~# uci add network device # =cfg070f15
cfg070f15
root@OpenWrt:~# uci set network.@device[-1].type=bridge
root@OpenWrt:~# uci set network.@device[-1].name=podman0
root@OpenWrt:~# uci set network.@device[-1].bridge_empty=1
root@OpenWrt:~# uci set network.podman0=interface
root@OpenWrt:~# uci set network.podman0.proto=static
root@OpenWrt:~# uci set network.podman0.device=podman0
root@OpenWrt:~# uci set network.podman0.ipaddr=10.88.0.1
root@OpenWrt:~# uci set network.podman0.netmask=255.255.0.0
root@OpenWrt:~# uci commit
root@OpenWrt:~# /etc/init.d/network restart

This also then exposes the interface for firewall setup; for example, to allow traffic from the podman containers to the world:

root@OpenWrt:~# uci add firewall zone # =cfg0edc81
cfg0edc81
root@OpenWrt:~# uci set firewall.@zone[-1].name='podman'
root@OpenWrt:~# uci set firewall.@zone[-1].input='ACCEPT'
root@OpenWrt:~# uci set firewall.@zone[-1].output='REJECT'
root@OpenWrt:~# uci set firewall.@zone[-1].forward='REJECT'
root@OpenWrt:~# uci add_list firewall.@zone[-1].network='podman0'
root@OpenWrt:~# uci add firewall forwarding # =cfg0fad58
cfg0fad58
root@OpenWrt:~# uci set firewall.@forwarding[-1].src='podman'
root@OpenWrt:~# uci set firewall.@forwarding[-1].dest='wan'
root@OpenWrt:~# uci commit
root@OpenWrt:~# /etc/init.d/firewall restart

root@OpenWrt:~# podman run --rm registry.fedoraproject.org/fedora-minimal curl https://openwrt.org/

[kirdesde]

A few of your points are already covered in that guide:

https://openwrt.org/docs/guide-user/virtualization/docker_host?s[]=podman#podman

[adelton]

I'm aware of that document and tried to follow it but AFAICS, the software has evolved and the document no longer describes a frictionless experience. So in a way I'm trying to figure out how to update that document with up-to-date expectations, besides potentially updating the packages.

For example, the opkg install list of packages is mostly not needed as the latest podman builds pull them as dependencies. The default /etc/containers/networks/podman.json is different and it is not clear why the document shows some modified example. There is no /etc/config/netavark there by default. Even if you create it and put the option driver 'none' there, it does not solve the iptables problem (on OpenWrt 24). And so on.

So there are parts there that don't need to be and there don't seem to be points documented that are on the critical path to a working setup.

[kirdesde]

Yes it's outdated in some points. The firewall driver "none" needs to set in the container.conf

https://forum.openwrt.org/t/podman-pod-briefly-available-then-unreachable/220561/11?u=kirdes