Unified Build June 2023 Status Update

[mmitche]

Quick Recap on Unified Build

The Unified Build Project is intended to improve two areas for .NET:

• Improve our upstream/downstream model – Partners are efficient and can do the same things Microsoft can.
• Reduce product build and servicing complexity – Tame .NET's incredibly complex infrastructure.

Most of the existing issues arise from the fact that .NET is a set of repositories that collectively ship together to form a coherent product. These repositories form a strong and deep dependency graph, and so Microsoft's NET build is not just a build of a single repository, but a highly complex series of builds and dependency updates orchestrated by tooling across this graph. Adding to this complexity, .NET also maintains a method of building (Source-Build) for use by Linux distro maintainers which involves no orchestration, but which is platform limited and does not meet Microsoft's needs.

Unified Build seeks to improve .NET's infrastructure and process by pulling Microsoft's current .NET build closer to Source-Build, without incurring the cost of development typically present in a single monolithic repository. Instead, we seek to continue to support development in the individual repositories. The individual repositories will insert source code into a "Virtual Monolithic Repository", which will be built across all platforms, including by Linux distro partners, to produce shipping outputs.

If you'd like to know more about the foundations and theory of Unified Build, please see the design documentation.

What has been accomplished so far/what are we working on?

Most of the Unified Build work in the .NET 8 timeframe is groundwork for major changes in .NET 9; design work, foundational infrastructure, and improvements to the sustainability of current systems. Highlights include:

• Overarching Unified Build design - The design work for much of Unified Build was completed in 2022 and can be seen here.
• Introduction of the VMR-lite - The VMR is a self-contained repository which can be used to build .NET 8 in the Linux source-build configuration. In .NET 7 and prior, this source layout was only generated as a source tarball during the dotnet/installer build. Creation of this repo required laying a lot of the infrastructural groundwork that will be required for .NET 9.
• Source-Build sustainability - Maintaining both methods of building (Microsoft and Linux source-build) has historically been quite expensive. The team identified several large areas where significant improvements could be made to repository validation and build logic, bringing Linux source-build closer to Microsoft's .NET build. This would help us reduce discrepancies that cause issues and detect other types of issues earlier in the development process. In many cases, issues like newly introduced dependencies, can be detected and dealt with at PR time in individual repositories. After months of work, the sustainability work has been completed, freeing the source-build team work time to work on improving other aspects of Linux source-build.
• Design work for handling SDK bands in the VMR - .NET has multiple SDK bands for each version of the runtime, which serve as a ship vehicle for updates to tooling. Handling this 1:many relationship has never been trivial, but requires careful thought in the VMR. This design work is in progress over at Managing SDK Bands and should be completed soon.

What's next on the horizon?

• Scenario testing - We plan to introduce a simple, self-contained test harness for running scenario tests against an installed product (an SDK, a runtime, etc.). As we expand the platforms that the source-build infrastructure supports, we need be able to verify that the built product continues to behave as expected. Scenario tests that execute against installed products give us a way to easily compare results between Microsoft's existing build and the source-built product.
• Vertical build improvements - We plan to make incremental improvements to Microsoft's build in .NET 8, especially in the runtime repo, to simplify it and bring it closer to build process required in .NET 9.
• Source-build multi-platform design - .NET uses various switches, msbuild properties, and build logic today to differentiate between Linux source-build and Microsoft's build and keep them running. Because source-build has only ever been widely used by distro maintainers on Linux, this infrastructure tends to imply information about the target platform. Expanding source-build to additional platforms such as Windows and Mac will require a reexamination of this design.
• Backflow tooling - The full .NET 9 VMR will flow both outputs of the .NET build, as well as source changes, back to individual development repositories. Designing and implementing this backflow tooling is important to the .NET 9 story.