[RFC] meson2hermetic in mesonbuild repo

[bphunguyen]

[RFC] meson2hermetic in mesonbuild repo
@gsingh408 and I and propose porting the meson2hermetic tool to the mesonbuild repository:

https://android.googlesource.com/platform/external/mesa3d/+/refs/heads/main/meson_to_hermetic/

It turns the meson build specification into hermetic build rules.

Where will we use this tool?
The use case will be initially Mesa3D integration in Fuchsia/Android, but we want to support a few other projects (QEMU, gfxstream_backend) as well.

The first two hermetic build systems we want to support are:

https://source.android.com/docs/setup/build
https://bazel.build/

Other major ones are buck2 (https://github.com/facebook/buck2) and (https://gn.googlesource.com/gn). Google, true to all the memes, maintains 3 public hermetic build systems and 1 major private one (Blaze).

Hermetic Build Systems: why not just use Meson + NinjaBuild?
Hermetic build systems tend to target a different use case (very large projects) than meson. The numbers are irresistible: all of Android can be built in ~30 minutes using the tricks allowed by hermeticization.

https://bazel.build/basics/hermeticity#benefits

The issue is these systems need a lot of resources to set-up and are not user friendly. Meson itself has benefits: a terrific API, specification, great for distros + developers. We want a bridge between these two worlds. This will allow meson.build files to be the source of truth for the various hermetic build systems. For example, llvm-project supports both GN and Bazel:

https://github.com/llvm/llvm-project/tree/main/llvm/utils/gn
https://github.com/llvm/llvm-project/tree/main/utils/bazel

What if we can add meson to llvm-project (they allow alternate build systems in the "peripheral” support tier), and then use meson2hermetic to convert to Bazel/GN? Maybe that could show the strengths of meson versus CMake in that project? That’s the sort of thing we can do with this tool, if it works well and consistently.

Current approach: It kind of works, but we can do better
Implementation-wise, our script is a two-step process:

Use a meson2python converter, since meson is mostly python-like:

https://android.googlesource.com/platform/external/mesa3d/+/refs/heads/main/meson_to_hermetic/generate_python_build.py

Overload the meson API in the python script with our custom functions.

(for example, see the “shared_library” overload)

https://android.googlesource.com/platform/external/mesa3d/+/refs/heads/main/meson_to_hermetic/meson_to_hermetic.py#858

Run the output python script multiple times
The meson arguments, toolchain (compiler, host machine), dependencies are given by a .toml file:

https://android.googlesource.com/platform/external/mesa3d/+/refs/heads/main/meson_to_hermetic/aosp.toml

Save state in each from each run
After all runs are complete, construct hermetic build rules

We did it this way since we just needed something quick and dirty, and it works well enough to unblock some work. We feel we can greatly improve the script by leveraging the internal meson APIs available in the meson repository. We want human-readable output from the tool, as if a knowledgeable human had hand-written the hermetic build tools.

Improvement Idea #1: Use Interpreter class from tool directly

A keen observer can see (1) can be removed almost certainly. It seems we can overload the meson API here:

https://github.com/mesonbuild/meson/blob/master/mesonbuild/ast/__init__.py
https://github.com/mesonbuild/meson/blob/master/mesonbuild/ast/interpreter.py#L119

Improvement Idea #2: Write another Backend

There is also the possibility of writing another backend. For example, a “Bazel” backend was added here:

https://android-review.googlesource.com/c/trusty/external/qemu-meson/+/3047774

That approach uses a shadow ninja build (which is avoidable), and it doesn’t do the multiple invocations of our current approach. But we think a “hermetic” meson backend is possible, if there is a way to invoke a backend multiple times with different arguments to construct the full build rules. The APIs used by meson introspection also look interesting.

Improvement Idea #3: Somehow access the Build class

We need the state (static libraries, include dirs, custom targets, paths in $SRC_DIR) from multiple builds to create the hermetic build rules. Idea: use Build class and export that?

https://github.com/mesonbuild/meson/blob/master/mesonbuild/build.py#L239

Dependencies
We use jinja2 for templates, ruff for the Linter, and tomlib for configuration. We'll transition to internal meson templates and meson uses pylint, but tomlib is core in python 3.11. We envisage this as an additional helper program:

https://mesonbuild.com/Contributing.html#external-dependencies

so we’ll probably keep that around.

Expected kLOC
2kLoC -> 5kLoC, the less the better.

Proposed intermediate state
Like cmake2meson.py, our hunch is that meson2hermetic.py should live in the {$MESON_SRC}/tools directory. It needs to access APIs from the ${MESON_SRC}/mesonbuild directory though.

Ideal end state
Maybe after the tool has reached sufficient maturity, we can make it an official meson API.

meson convert [mesa-aosp.toml, mesa-fuchsia.toml, qemu.toml] ${SRC_DIR} ${TARGET_DIR}

HELP NEEDED FOR DESIGN: meson internals
We don’t understand the meson codebase well enough to know the best approach, but feel it’s doable given Meson’s current abstractions.

We would like your advice on how an experienced Meson developer would implement the tool. We can add the implementation, tests, make sure the tool is actively used (https://autoroll.skia.org/r/mesa-android) etc. – but we need your feedback on the design.

CC: @eli-schwartz @dcbaker @bonzini