Bazel: goals, roadmap, expectations

[vitvakatu]

We wanted to provide additional context for the work of moving our build system to Bazel, answer common questions, provide some expectations, and explain our long-term goals. So here it is.

Goals

When transitioning to Bazel, we are aiming at the following goals:

• Get rid of the slow and complicated Rust build tool. It is compiled, so we pay for each CI run at compilation time, and only a limited number of people are capable of changing it. With this change, we want to reduce CI times and involve more developers in configuring the build.
• Simplify CI configuration. With the current setup, we have multiple workflows, some of them auto-generated by the Rust build tool and some manually written. We manually handle “skip if not changed” checks and overall have a pretty fragile setup trying to balance CI speed and robustness. Auto-generation of CI workflows is mostly a good thing, but it is done by the Rust build tool; see the previous point.
• We want a single configuration system (at least at the top level) that manages the whole product across different platforms and programming languages. Being able to define dependencies between modules in different languages allows us to avoid complex multi-staged builds with “glue scripts”, as we have now, e.g. with Enso parser codegen. The intended result is again simplifying CI configuration and involving more developers in configuring the build.
• Building the whole application and running any set of tests should be possible locally, on all platforms, preferably with a single command. There should be no need to manually setup required system dependencies. Running only tests affected by code changes is an additional goal, helping to reduce CI load.

Overall, we want to simplify and speed up our current workflows while mostly preserving user-facing (developer-facing) interactions.

Current state of things

As of 2025-01-29, we partially integrated Bazel into the build process of the GUI parts of the main repository. This was done in #11337 and a couple of subsequent PRs. For this integration, we tried to preserve the existing workflows as much as possible and not break anything related. We mostly succeeded, but not completely.

Bazel is used internally when you run pnpm install, as part of the postinstall hook. Bazel (via Bazelisk) is installed as a pnpm dependency and then used as bazel run //:write_all.
This postinstall hook is used to generate a couple of necessary artifacts for other stages of the build, namely:

1. Generate icon metadata so that we can autocomplete icon names in the code
2. Generate JS code bindings for our Rust-based parser
3. Compile and package the rust-ffi WASM module used primarily for the parser on the frontend.

Keep in mind that this postinstall script is not necessary for the Bazel build itself. These actions are only needed for the “legacy” build system to continue working without breakages. Eventually, we will remove the postinstall hook.

Because of Bazel, you technically do not need to install Rust or any other native dependency when executing pnpm install, pnpm and Bazel will manage everything automatically. However, we continue to use our Rust-based build system for other stages of the build, and it was largely untouched. As a long-term goal, every build stage will be managed by Bazel configuration.

Besides that, we also support building GUI distribution entirely by Bazel. As an equivalent of running pnpm build:gui, you can use bazel build //app/gui:dist. The build is checked by a separate CI job, although we don’t check if it runs fine. You can also run Nodejs-based ydoc server by using bazel run //app/ydoc-server-nodejs:server. With the ydoc server and project manager running separately, we checked that the GUI distribution built by Bazel is working well.

To sum it up:

1. Bazel is implicitly used in the current build process already.
2. We can build a fully working GUI distribution by Bazel alone.
3. Engine distribution and a lot of dev workflows and CI jobs have not been translated to Bazel yet.

Our plans and hopes

We decided to commit to a full transition towards Bazel as a build system for the whole project. We expect that this effort will take some undetermined time, and we want to make it as non-disturbing as possible while also delivering user-facing features for the product.

We believe that using Bazel will simplify CI configuration, speed up builds and all development workflows, and make our builds and tests more stable and predictable.

So, we want, without any specified timeline:

1. Transition Engine distribution to Bazel.
2. Transition building Electron package to Bazel
3. Transition remaining actions like linting, running tests, and local dev-server to Bazel
4. Get rid of the remaining parts of the Rust-based build tool (./run script)
5. Experiment with remote builds and alternative CI implementations (like https://aspect.build) along the way

Environment variables management

Bazel is a build system focused on the heavy reuse of cached dependencies and explicit dependency declaration for each build step.
This means that one of the most important changes is related to using build-time environment variables to configure the application. Bazel isolates the environment when running build steps, so it is hard to pass a particular build-time environment variable to the build. To mitigate this issue, we slightly reworked our infrastructure for build-time environment variables.

How it works

import.meta.env access is discouraged in the code. Instead, we have a single application-wise $config object (see app/gui/src/config.ts). Using envPrefix feature of Vite we populate this object with environment variables starting with ENSO_IDE_ prefix. Vite also automatically loads .env files from app/gui directory.

We have a Git-committed .env file at app/gui and expect you to provide .env.local file with sensitive environment variables used for dev builds. On CI, we simply expose environment variables to the environment for now. See CONTRIBUTING.md for details.

It is still possible to expose needed environment variables with ENSO_IDE_ prefix in your shell environment, but it is discouraged, as it won’t work for Bazel builds. Remember, Bazel isolates the shell environment and only uses .env files to provide Vite with input. While this still works currently for “legacy” builds, as we transition to Bazel your code will break. Don’t do that.

How can I override particular build-time variable for local build?

Change/add an entry in your app/gui/.env.local. If you want to explicitly unset a certain variable, set it to empty string (ENSO_IDE_WHATEVER=). Avoid modifying .env file, as it is committed to the repo.

If you want to override many variables, replace the .env.local file. We have an active bug report that selecting the custom .env.[mode] file by CLI flag does not work for now. See #12181.

How can I use the build-time environment variable in the code?

Use $config.whatever. To check for an unset variable, look both for undefined and an empty string. JS has nice implicit coercion to the boolean in this case. (if ($config.variable) { doSomething($config.variable) } else { doNothing() })

How can I add a new build-time environment variable?

Add it to the .env file and commit changes; set it to an empty string or placeholder if sensitive. Add new field to $config object. If necessary, override locally using .env.local. That’s all.

What about runtime environment variables?

We usually do not use runtime environment variables in the application, but if necessary, just use process.env on the Electron side. No changes here.

CI, past, present and future

We currently use a mix of GitHub-hosted and self-hosted machines to run our CI jobs. We introduced self-hosted runners mostly to have a persistent cache across builds because rebuilding the Rust codebase from scratch without any caches was too costly. Because of this, our self-hosted runners live in persistent docker containers, which are restarted rarely.

This means that jobs on self-hosted runners are not entirely isolated from one another because they share the same persistent filesystem. There is no issue with concurrent access — each runner executes a single job at a time. For now, we decided to disable network caches for Bazel on self-hosted machines because downloading them over the internet takes too much time. Instead, we rely on the fact that we don’t restart the runners, and we reuse local filesystem caching that is automatically managed by Bazel on a per-runner basis. We feel it is an acceptable intermediate solution.

With Bazel, we have multiple different routes available:

• We can eliminate self-hosted runners and use GitHub runners only. With effective caching (downloading cache on Github runners takes only a few seconds), there is no need for self-hosted runners aside from price and limit considerations.
• We can use a third-party CI system like https://aspect.build to reap all the benefits of Bazel
• We can combine dozens of our CI jobs into a single job, avoiding running common build steps multiple times on different runners. For example, currently, almost every job runs pnpm install and then compiles the Rust build tool. Some jobs even spend time installing the Rust compiler from scratch each time. Instead, a single job can build all necessary targets, run needed tests, and prepare artifacts. We will need additional tools to improve the discoverability of individual step results, but again, Aspect.build can help with that.
• We can avoid awkward “rebuild what changed” workarounds by using Bazel to manage this. If we specify the dependencies between our targets, Bazel will be able to rebuild only the needed parts for the list of needed outputs.

All these ideas require some research and some time to implement, but they can potentially simplify our CI configuration drastically.

Known issues, where to look for progress, what now

If you want to see what Bazel-related issues were already reported, or what ideas we want to research — use this task: #10145

We are aware of one particular issue in Bazel builds on Windows that is too complicated to fix on our side. Because Windows lacks proper build isolation, moving and renaming typescript files can accidentally trigger cryptic build issues related to incorrect import statements in the code. If you see this issue on Windows runners only, and everything is fine on other platforms — feel free to do a clean build. The issue can be fixed by removing bazel output artifacts (bazel-* directories) from the repository root.

Among our most essential tasks in the near future is transitioning Engine distribution to Bazel.

While we continue working on Bazel, there can be different bugs and regressions. While understanding the importance of working CI and local builds, we like to note that we will not disable, rollback, or in any way hinder Bazel parts that were already integrated. We want to move forward reasonably slowly, but we will move forward, not backward. If you encounter any issue, let’s discuss it, prioritize it, and solve it, as with any other product issue we face.

[JaroslavTulach]

Rust build tool. It is compiled, so we pay for each CI run at compilation time
... almost every job runs pnpm install and then compiles the Rust build tool.

Now, when we have the amazing Bazel caching capabilities: could we cache the built bits of the Rust build tool to begin with?

• that would avoid compilation of Rust at the beginning for the CI run
• the cache would be very effective - nobody wants to touch the Rust build code - so it is very stable
• people say they don't want to maintain the Rust build tool, but
• this would be just Bazel code around it - and Bazel is the future, isn't it?

this effort will take some undetermined time,
... as non-disturbing as possible
... delivering user-facing features for the product

The great thing on caching bits of the Rust build tool is:

• we get some of the Bazel benefits immediately (not in an undetermined time)
• the build gets faster (a feature has been delivered)
• Bazel must have support for Rust - so the change should be straightforward (it is non-disturbing)

When we have the Bazel hammer, why not use it?

[JaroslavTulach]

• there is no need for self-hosted runners aside from price and limit considerations.

;-)