Using sysbox with Bottlerocket

[lox]

We've been making heavy use of Sysbox (https://www.nestybox.com/sysbox) for CI/CD jobs to allow docker-in-docker without needing privileged containers.

It seems like with the new Kernel versions that are landing this should be relatively straightforward to make work in Bottlerocket.

Has anyone done this? Would there be interest in me contributing a package if not?

[ctalledo]

Hi, I am one of the developers of Sysbox.

It seems like with the new Kernel versions that are landing this should be relatively straightforward to make work in Bottlerocket

In particular Sysbox uses the kernel's ID-mapped-mounts feature, introduced in upstream kernel 5.12. If Bottlerocket includes that, then likely Sysbox should work on it.

[stmcginnis]

Hey @lox - thanks for bringing this up!

This is an interesting use case and there are few discussion ongoing about it. The current work focusing around this is being tracked in #2163.

It would be great if you could check that out and add any feedback about your particular use case.

I would also welcome you to join our bi-weekly community meeting if you'd like to discuss some of these plans with the community. You can find information about our community meetings here: https://www.meetup.com/bottlerocket-community/

Thanks!!

[lox]

No problems! I had a look at #2163, that looks like it might be related to containers with multiple users (perhaps for dropping privileges from root to a less-privileged user?). That sounds like an overlap with the ID-mapped-mounts feature that Sysbox uses, but that's not what we're looking to use here. (Unless I've misunderstood and it provides rootless containers some other way)

In CI/CD, you frequently need access to a docker daemon to create containers for testing (for instance to created an ephemeral MySQL + Redis environment to run a test suite agains). Prior to Sysbox, the "state of the art" here was to run either "Docker in Docker" (which requires containers to be run with --privileged which gives them root access to the host), or by passing the docker socket into the containers (which again, gives root access to the host). Sysbox is an OCI runtime that allows docker-in-docker without the --privileged flag (amongst other security hardenings). Here is a good blog post from one of the original docker authors that explains it (and calls out Sysbox).

What we'd like to do is to use Bottlerocket to run a Buildkite Agent on Bottlerocket that runs each CI/CD job in a Sysbox Container on the Bottlerocket host, each with it's own Docker Daemon running internally.

Hope that helps context wise!

[gadiener]

Hi @lox, were you able to run Sysbox on Bottlerocket? I have a similar use case with Jenkins+DinD on EKS and would be interested in hearing more about your results

[arnaldo2792]

I think one experiment worth trying is this:

• Launch an unprivileged pod that runs DinD in a Bottlerocket host, with user.max_user_namespaces > 0, and configure Docker to use sysbox-runc
• Make sure that newuidmap is installed in the container that runs the docker daemon, so that the user mapping works
• Create a container through DinD

According to this, sysbox will create the user mappings if none were provided. If the container runs, then there is some confidence that this setup will work. But keep in mind that the pod in Bottlerocket won't be running in its own user namespace, since more work is needed in k8s and Bottlerocket to support user namespaces and UID/GID mappings.

I'm posting this experiment here so that anybody can give it a try.