Error on macOS aarch64

[chadbrewbaker]

Built from commit b7e589f .

xcodebuild
./build/Release/xhyve -h
zsh: killed   ./build/Release/xhyve -h

Is this expected behavior or a bug?

[chadbrewbaker]

This is upsetting. https://forums.macrumors.com/threads/if-you-disable-sip-all-you-ios-apps-will-stop-working-on-your-m1.2269661/

Turning off SIP just to run dtruss hoses apps permanently. You would have to link with a tracing glibc to get system calls.

[chadbrewbaker]

ls /Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk//System/Library/Frameworks/Kernel.framework/Versions/Current/Headers/arm64 
machine_cpuid.h		machine_machdep.h

These seem to be the arm64 headers to fix src/vmm/x86.c ?

[chadbrewbaker]

https://developer.apple.com/documentation/hypervisor/apple_silicon

[chadbrewbaker]

@jonmasters got Fedora to boot on M1 https://twitter.com/jonmasters/status/1331137023386800128

[741g]

That's with Virtualization.framework, which is the easier-to-use library versus Hypervisor.framework (and may be more appropriate even?)

The tweet references https://github.com/KhaosT/SimpleVM

[jonmasters]

Right. I was trying to get something quick and dirty up and running, so I used the Virt framework. A full solution built using the Hypervisor framework would be nice to have since that would provide an ability to add/remove hardware and change the machine model, among many other things.

[fayep]

To answer @chadbrewbaker original question. The build generates an x86_64 binary and Rosetta won’t work for virtualization which was made clear by Apple. This won’t work for arm64 until it is ported and if you are interested you should look at the architecture specific code in FreeBSD-UPB/freebsd:bhyvearm64-rebase for inspiration. While following this paper: https://papers.freebsd.org/2019/bsdcan/elisei-bhyvearm64_cpu_and_memory_virtualization_on_armv8.0_a/

[741g]

I'm working on a stack of Hypervisor.framework patches here: https://android-review.googlesource.com/c/platform/external/qemu/+/1510277

[brandonros]

I'm working on a stack of Hypervisor.framework patches here: https://android-review.googlesource.com/c/platform/external/qemu/+/1510277

@741g looks like that got merged. what are the next steps? :)

https://gist.github.com/osy86/10d0ac0210f0eb99474ae31cf8d6dd9e#file-windows-10-arm-utm-zip for anybody else that lands on this

[741g]

You can pretty much follow those and incorporate into the xhyve implementation. It was quite straightforward compared to x86 support and as long as you have virtual firmware for setting up memory maps and routing interrupts, it should be doable. The key parts are:

1. Setting up an arm64 set of source files in xhyve/src for storing register state along with timer and GIC state. You can use the qemu version as a reference. You'll probably need a different set of virtual firmware sources. You can use the virt board (https://android.googlesource.com/platform/external/qemu/+/refs/heads/emu-master-dev/hw/arm/virt.c) as a reference. You'll need to implement something that stores information for memory maps, irqs, etc. and implements the state for GIC and vtimer.

2. I notice there's a lot of x86-isms still present in the source, such as vmm_ioport.c, vmm_util.c, vmm_lapic.c, etc. vmm.c will need different handlers for ARM64 style exits such as DABT and WFx in addition to Hypevisor.framework's VTIMER. (see the code for reference).

3. First, try getting something basic running that uses hv_vcpu_run, hv_vm_create, hv_vm_map that follows how it's being used in xhyve, while aborting on all exits, similar to https://android.googlesource.com/platform/external/qemu/+/c17eb6a3ffd50047e9646aff6640b710cb8ff48a

You'll need to ensure that

• Hypervisor.framework's PC register is pointed to the same beginning address specified by virtual firmware in xhyve
• Memory maps are all set up
• Register sync up when guest exits and pre-guest-enters is working. We found https://android.googlesource.com/platform/external/qemu/+/refs/heads/emu-master-dev/target/arm/hvf.c#699 to be a sufficient set of registers.

4. Implement VM exit handling

There will be these types of Hypervisor.framework exits that you need to handle for ARM64:

• EL2 Exception (https://android.googlesource.com/platform/external/qemu/+/refs/heads/emu-master-dev/target/arm/hvf.c#1439), which includes the following subclasses:
  • WFx (Wait for Interrupt)
  • Guest Abort (Data or Instruction Abort) for MMIO
  • SMC
  • system register access
• VTimer access (https://android.googlesource.com/platform/external/qemu/+/refs/heads/emu-master-dev/target/arm/hvf.c#1595)
  • This happens when the guest sets a timer but doesn't really have much to do after that and wants to be woken up later on

5. Implement kicking the vcpus out of running the guest (https://android.googlesource.com/platform/external/qemu/+/cfaec60e391f549fed43736aa09fa670416b6bdf)

• here: https://github.com/machyve/xhyve/blob/master/src/vmm/vmm.c#L1689
  • you'll need to use hv_vcpus_exit() API