make-disk-image: add binfmt emulation

[saviosg]

Now it's possible to build images for other architectures with native speed using KVM.
Binfmt emulation is only needed for the bootloader installation step, other steps use native binaries.

To use it, set imageBuilder.enableBinfmt to true, and pass a new nixpkgs instance to the image builder, with the system set to the running machine.

Example, building an aarch64 image on an x86_64 host:

nixpkgs.lib.nixosSystem {
  system = null;
  modules = [
    {
      nixpkgs.buildPlatform = "aarch64-linux";
      nixpkgs.hostPlatform = "aarch64-linux";
    }
  ];
}

...

imageBuilder.enableBinfmt = true;
imageBuilder.pkgs = nixpkgs { system = "x86_64-linux"; };

[iFreilicht]

Very cool! Could you try to add a test for this as well? No need to test all permutations, aarch64 on x86_64 would be sufficient.

[phaer]

Very nice feature, thank you!

Don't want to block, but I think copying the magic numbers and such from nixpkgs and duplicating them in diskos source might not be the best course of action long-term.

Maybe nixpkgs could expose them via a non-visible readOnly option or so instead?

[Mic92]

I think copying might be fine. Having a new CPU architecture is a rare event.

[saviosg]

I didn't have time to work on this PR the last few weeks, but I'm working on adding tests and minor fixes soon.

Regarding the duplicate code, I agree it's not the most ideal approach, but if that's ok we can do it this way for now and open a separate PR in nixpkgs to export the binfmt code to the lib.

[Nebucatnetzer]

Thank you a lot for your work on this.

[danjujan]

Thank you for this PR! I tried it out but still had a few issues with it.
Had to set imageBuilder.kernelPackages to e.g. inputs.nixpkgs.legacyPackages.x86_64-linux.linuxPackages_latest otherwise x86_64 qemu would try to boot an aarch64 kernel in my case.

binfmtSetup should also be executed in the diskoImagesScript section.

And in the formatting step it failed to create btrfs subvolumes.

+ btrfs subvolume create /tmp/tmp.lEcwmBFds7/@
ERROR: Could not create subvolume: Inappropriate ioctl for device```

[saviosg]

You are right. I'm going to update the PR with the things you mentioned.

I haven't run into the Btrfs issue, I also use Btrfs but only the root subvolume. You could open a separate issue and include your disko config.

[Mic92]

You are right. I'm going to update the PR with the things you mentioned.

I haven't run into the Btrfs issue, I also use Btrfs but only the root subvolume. You could open a separate issue and include your disko config.

Shouldn't it be possible to run disko with the native system and only run the installer with the target arch? Filesystems will use a lot of uncommon ioctls so I don't think this will be the last one.

[danjujan]

You are right. I'm going to update the PR with the things you mentioned.
I haven't run into the Btrfs issue, I also use Btrfs but only the root subvolume. You could open a separate issue and include your disko config.

Shouldn't it be possible to run disko with the native system and only run the installer with the target arch? Filesystems will use a lot of uncommon ioctls so I don't think this will be the last one.

The following diff works for me. Should I open a seperate PR or do we want to commit it directly to this one?

diff --git a/lib/make-disk-image.nix b/lib/make-disk-image.nix
index 7281a16..b4a0b7d 100644
--- a/lib/make-disk-image.nix
+++ b/lib/make-disk-image.nix
@@ -51,6 +51,18 @@ let
       }
     ];
   };
+  systemToInstallNative = if binfmt.systemsAreDifferent then extendModules {
+    modules = [
+      cfg.extraConfig
+      {
+        disko.testMode = true;
+        disko.devices = lib.mkForce cleanedConfig.disko.devices;
+        boot.loader.grub.devices = lib.mkForce cleanedConfig.boot.loader.grub.devices;
+        nixpkgs.hostPlatform = lib.mkForce pkgs.stdenv.hostPlatform;
+      }
+    ];
+  }
+  else systemToInstall;
   dependencies = with pkgs; [
     bash
     coreutils
@@ -88,7 +100,7 @@ let
     ln -sfn /proc/self/fd/2 /dev/stderr
     mkdir -p /etc/udev
     mount -t efivarfs none /sys/firmware/efi/efivars
-    ln -sfn ${systemToInstall.config.system.build.etc}/etc/udev/rules.d /etc/udev/rules.d
+    ln -sfn ${systemToInstallNative.config.system.build.etc}/etc/udev/rules.d /etc/udev/rules.d
     mkdir -p /dev/.mdadm
     ${pkgs.systemdMinimal}/lib/systemd/systemd-udevd --daemon
     partprobe
@@ -98,10 +110,11 @@ let
     ${lib.optionalString diskoCfg.testMode ''
       export IN_DISKO_TEST=1
     ''}
-    ${lib.getExe systemToInstall.config.system.build.destroyFormatMount} --yes-wipe-all-disks
+    ${lib.getExe systemToInstallNative.config.system.build.destroyFormatMount} --yes-wipe-all-disks
   '';
 
   installer = lib.optionalString cfg.copyNixStore ''
+    ${binfmtSetup}
     # populate nix db, so nixos-install doesn't complain
     export NIX_STATE_DIR=${systemToInstall.config.disko.rootMountPoint}/nix/var/nix
     nix-store --load-db < "${closureInfo}/registration"