Kubernoodles

Kubernoodles is a framework for managing custom self-hosted runners for GitHub Actions in Kubernetes at the enterprise-wide scale. The design goal is to easily bootstrap a system where customized self-hosted runners update, build, test, deploy, and scale themselves with minimal interaction from enterprise admins and maximum input from the developers using it.

This is an opinionated reference implementation, designed to be taken and modified to your liking. I use this to test GitHub Actions on my personal account, GitHub Enterprise Cloud (SaaS) or GitHub Enterprise Server (self-hosted) from Docker Desktop, a Raspberry Pi cluster for arm64, a managed Kubernetes provider, and other random platforms as needed. Your implementation may look wildly different, etc.

❓ Are you a GitHub Enterprise admin that's new to GitHub Actions? Don't know how to set up self-hosted runners at scale? Start here!

Pull requests welcome! ❤️

Setup

The admin introduction walks you through some key considerations on how to think about implementing GitHub Actions at the enterprise scale, the implications of those decisions, and why this project is generally built out the way it is.

The admin setup is a mostly copy-and-paste exercise to get a basic deployment up and going.

The customization guide has a quick writeup and links to learn more about the ways you can customize things to your needs.

Tips and tricks has a few more considerations if things aren't quite going according to plan.

Choosing the image(s)

There are currently 5 images that are "prebuilt" by this project, although you can certainly use others or build your own! All images assume that they are ephemeral. If you're copy/pasting out of the deployments, you should be set ... provided you give it the right repository/organization/enterprise to use!

image name	base image	CVE count (crit/high/med+below)	archs	virtualization?	sudo?	notes
ubi8	ubi8-init:8.10	0/4/552	x86_64 arm64	❌	❌	n/a
ubi9	ubi9-init:9.5	0/4/547	x86_64 arm64	❌	❌	n/a
rootless-ubuntu-jammy	ubuntu:jammy	0/4/170	x86_64 arm64	rootless Docker-in-Docker	❌	common rootless problems
rootless-ubuntu-numbat	ubuntu:numbat	0/4/86	x86_64 arm64	rootless Docker-in-Docker	❌	common rootless problems
wolfi:latest	wolfi-base:latest	0/1/0	x86_64 arm64	❌	❌	n/a

Note

CVE count was done on 16 November 2024 with the latest versions of grype and runner image tags.

Design goals and compromises

There are a few assumptions that go into this that aren't necessarily true or best practices outside of an enterprise "walled garden". Being approachable and readable are the most important goals of all code and documentation. As a reference implementation, this isn't a turn-key solution, but the amount of fiddling needed should be up to you as much as possible. Links to the appropriate documentation, resources to learn more where needed, and explanations of design choices will be included!

Co-tenanted business systems tend to have small admin teams running services (like GitHub Enterprise) available to a large group of diverse internal users. That system places a premium on people-overhead more than computer-overhead. The implication of that is an anti-pattern where there are larger containers capable of lots of different things instead of discrete, "microservices" type containers.

Moving data around locally is exponentially cheaper and easier than pulling data in from external sources, especially in a larger company. Big containers are not scary if the registry, the compute, and the entire network path is all within the same datacenter or availability zone. Caching on-site is important to prevent rate-limiting by upstream providers, as that can take down other services and users that rely on them. This also provides a mechanism for using a "trusted" package registry, common in enterprise environments, using an .env file as outlined here.

Sources

These are all excellent reads and can provide more insight into the customization options and updates than are available in this repository. This entire repository is mostly gluing a bunch of these other bits together and explaining how/why to make this your own.

• GitHub's official documentation on hosting your own runners.
• Kubernetes controller for self-hosted runners, on GitHub, is the glue that makes this entire solution possible.
• Docker image for runners that can automatically join, which solved a good bit of getting the runner agent started automatically on each pod, write up and GitHub.
• GitHub's repository used to generate the hosted runners' images (GitHub), where I got the idea of using shell scripts to layer discrete dependency management on top of a base image. The software scripts are (mostly) copy/pasted directly out of that repo.

Learn more

• Don't know what the whole Kubernetes thing is about? Here's some help:
  • The Kubernetes Aquarium
  • The Cloud Native Computing Foundation's book, The Illustrated Children's Guide to Kubernetes
  • The official tutorial covering the basics of what Kubernetes is and how it works
  • What helped me to understand this whole concept shift is to think that Kubernetes is to containers as KVM/vSphere/Hyper-V is to virtual machines. It's probably not a perfect metaphor, but it helped. 😄
• Want to see a whole bunch of other ways to solve this problem? You should check out Awesome Runners for a curated list and amazing matrix comparison of all sorts of other self-hosted runner solutions.
• Even if this is 100% on-premises, many of these antipatterns for cloud applications are very relevant to the architecture of CI at scale and these are all well worth the time to read.
• Rootful versus rootless containerization in Podman is a bit different than in Docker. Learn more at RedHat's Enable Sysadmin blog post.

Dependencies of note

• actions-runner-controller
• Helm
• Yelp dumb-init
• Docker engine and Docker Compose for Debian-based images
• Podman, Buildah, and Skopeo for the RedHat-based images
• actions/runner is the runner agent for GitHub Actions

Note

GHES users prior to 3.9, please navigate back to tag v0.9.6 (release) for the APIs that'll work for you. ❤️