vm_image

Creating a VM using an AppJail image

vm-bhyve is a great choice for managing VMs on a FreeBSD box, it is easy, fast and very simple but powerful. However, there is one thing that is beyond the scope of this tool, which is to install a FreeBSD OS in "scripted-mode". Fortunately, it is very easy, plus we will use a custom AppJail image with everything needed to run the VM without problems.

Before continuing, we need vm-bhyve and appjail-reproduce. This last tool is necessary to define how the image will be built, which includes all the configuration that our VM will have.

I'm using the following versions:

# vm version
vm-bhyve: Bhyve virtual machine management v1.6-devel (rev. 106001)
# appjail version
3.5.0+54fb7f7e1e6260e31cc5a4c881bb162610ec68d4
# appjail-reproduce -v
0.4.0+211568647b45688249d0b4d0717bda93be2765fe

Read the documentation for both tools to find out how to configure them exactly as you want. I only mention ~/.config/appjail-reproduce/config.conf, the configuration file for appjail-reproduce, which includes at least two parameters:

~/.config/appjail-reproduce/config.conf:

COMPRESS_ALGO=zstd
BEFORE_MAKEJAILS="/root/Reproduce/main.makejail"

The first parameter COMPRESS_ALGO specifies the algorithm to compress the image. zstd is a great choice because of a good ratio and good performance, but you can choose the default, xz if you have the power to use it.

The second parameter BEFORE_MAKEJAILS which is a space separated list of Makejails to include before including the Makejail we will run later. main.makejail includes another Makejail called pkg.makejail which copies a pkg.conf(5) to use the latest branch.

/root/Reproduce/main.makejail:

INCLUDE pkg.makejail

/root/Reproduce/pkg.makejail:

CMD mkdir -p /usr/local/etc/pkg/repos
COPY /usr/local/etc/pkg/repos/Latest.conf /usr/local/etc/pkg/repos

/usr/local/etc/pkg/repos/Latest.conf:

FreeBSD: {
  url: "pkg+https://pkg.FreeBSD.org/${ABI}/latest",
  mirror_type: "srv",
  signature_type: "fingerprints",
  fingerprints: "/usr/share/keys/pkg",
  enabled: yes
}

Now we will create a project for appjail-reproduce.

mkdir -p ~/.reproduce/projects/vm

This project will have several files. The most relevant are Makejail, toremove.lst and reproduce.conf which are used by appjail-reproduce. The rest of the files are used by the Makejail. You can put in your project any file you want to tune the VM to your needs.

reproduce.conf:

tags: 14.2/14.2-RELEASE
arch: amd64
remove_rc_vars: defaultrouter
release: vm
args: hostname timezone
14.2.args.hostname: vm.lan
14.2.args.timezone: America/Caracas

Makejail:

ARG hostname
ARG timezone

OPTION start
OPTION overwrite=force
OPTION virtualnet=:<random> default
OPTION nat
OPTION resolv_conf=resolv.conf
OPTION type=thick
OPTION copydir=files
OPTION file=/boot/loader.conf
OPTION file=/etc/fstab
OPTION file=/etc/rc.conf
OPTION file=/etc/sysctl.conf
OPTION tzdata=$timezone

PKG neovim \
    terminfo-db \
    neofetch

SYSRC "hostname=${hostname}"

toremove.lst:

-f var/log/* 2> /dev/null || :
-f var/cache/pkg/*
-f var/run/* 2> /dev/null ||

resolv.conf:

nameserver 1.1.1.1
nameserver 1.0.0.1

files/boot/loader.conf:

nvme_load="YES"
if_bridge_load="YES"
bridgestp_load="YES"
kern.racct.enable=1

files/etc/fstab:

/dev/nda0p3        /           ufs         rw        1        1
/dev/nda0p2        none        swap        sw        0        0

files/etc/rc.conf:

# HOSTNAME
hostname=""

# Clear /tmp
clear_tmp_enable="YES"

# SYSLOGD
syslogd_flags="-ss"

# DHCP
ifconfig_vtnet0="DHCP"

# SSHD
sshd_enable="YES"

# MOUSED
moused_nondefault_enable="NO"

# DUMPDEV
dumpdev="NO"

# fsck(8) -y
fsck_y_enable="YES"

files/etc/sysctl.conf:

# A bit of hardening
security.bsd.see_other_uids=0
security.bsd.see_other_gids=0
security.bsd.see_jail_proc=0
kern.randompid=1

# Allow packet filtering in if_bridge(4)
net.link.bridge.pfil_member=1
net.link.bridge.pfil_bridge=1

Note a few things: I have created a release with appjail-fetch(1) called vm. This release has the base.txz and kernel.txz components. The latter is important for the VMs to work correctly.

# appjail fetch www -v 14.2-RELEASE vm base.txz kernel.txz
...
# appjail fetch list
ARCH   VERSION       NAME
amd64  14.2-RELEASE  default
amd64  14.2-RELEASE  vm

Once the project is defined, you can create the image with a single command.

# appjail-reproduce -fb vm
[ info  ] Started at Fri Dec 27 16:49:19 -04 2024
[ info  ] [vm]:
[ info  ] Logs: /root/.reproduce/logs/vm/2024-12-27_16h49m19s.log
[ info  ] > [vm] (osarch:amd64, osversion:14.2-RELEASE, tag:14.2):
[ info  ] Executing Makejail: jail:reproduce_24dd6371-80df-47a3-88fe-40c52a9de208, release:vm, args:"--hostname" "vm.lan" "--timezone" "America/Caracas"
[ info  ] Execution time: 00:01:39
[ info  ] Removing: rm -f var/log/* 2> /dev/null || :
[ info  ] Removing: rm -f var/cache/pkg/*
[ info  ] Removing: rm -f var/run/* 2> /dev/null || :
[ info  ] Removing rc variable: defaultrouter
[ info  ] Exporting vm
[ info  ] Export time: 00:00:48
[ info  ] ---
[ info  ] Ended at Fri Dec 27 16:52:09 -04 2024
[ info  ] Build time: 00:02:50
[ info  ] Hits: 1
[ info  ] Errors: 0
# appjail image metadata info vm
appjail image metadata info vm
Name            :    vm (14.2)
Build on        :
  - amd64
Image           :    amd64
  - SHA256 = b8633ca3a4bff8da706bbade7159b256dab7cd635c93f76a41039b5f657c2d6e
  - SIZE = 1683642096 (1.56GiB)
  - TIMESTAMP = Fri Dec 27 16:51:37 2024
Installed       :
  - /cloud-machine/appjail/cache/images/vm/14.2-amd64-image.appjail

The image was successfully created, so we proceed to create the VM (without starting it), create the partitions for the virtual disk and unpack the jail directory inside the image into the mounted root partition of the VM.

vm create -t CS0 -s 20G vm
mdconfig -at vnode /cloud-machine/vm/vm/disk0.img
gpart create -s gpt md0
gpart add -t freebsd-boot -s 512k -a 1m md0
gpart add -t freebsd-swap -s 2G -a 1m md0
gpart add -t freebsd-ufs -a 1m md0
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 md0
newfs -Ujt /dev/md0p3
mount /dev/md0p3 /mnt
tar -C /mnt --strip-components=2 -xf /cloud-machine/appjail/cache/images/vm/14.2-amd64-image.appjail ./jail/
umount /mnt
mdconfig -du 0

The template for vm-bhyve that I used, CS0, is as follows:

/cloud-machine/vm/.templates/CS0.conf:

loader="bhyveload"
cpu=1
memory=256M
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"

The last step is to start the virtual machine and access it.

vm start vm
vm console vm

That's all!