In-process runfile tree creation from SymlinkTreeAction fails for some consecutive builds

[Andrius-B]

Description of the bug:

We are facing an interesting hermeticity issue when creating runfiles locally. It is solvable by running bazel clean but preferably should be addressed. The premise of the bug is that the in-process runfile tree creation does not handle all cases of the previuos runfile tree.
With the flags

build --enable_runfiles
build --experimental_inprocess_symlink_creation

the in-process symlink creation will be used and thus the bug can be triggered.
In the example reproduction case, we have a target output that can be produced from two different inputs depending on a flag:

config_setting(
    name = "use_mode",
    flag_values = {"//:use": "True"},
)

make_folder(name = "make_folder")

use_folder(
    name = "use_folder",
    srcs = glob(["prepared/**/*"]),
)

output(
    name = "output",
    src = select({
        "//:use_mode": ":use_folder",
        "//conditions:default": ":make_folder",
    }),
)

then the output target is built two times by using different input rules. The make_folder rule produces the files by running a shell script and the use_folder produces the same files by globbing them from the sources. Both of these inputs are linked using runfiles to the same output path albeit a little differently, with:

• make_folder using ctx.runfiles(symlinks = {"lib": directory})
• use_folder using ctx.runfiles(symlinks = symlinks) with a complete dict for each file.

The bug is triggered by building the output target twice in a row using different configurations, therefore forcing the runfiles tree builder into an edge case.
Depending on the build oder, the error is slightly different:

$ ./repro1.sh
[...]
ERROR: /private/tmp/repr/BUILD.bazel:21:7: Creating runfiles tree bazel-out/darwin_arm64-fastbuild/bin/exec_output.runfiles failed: java.io.IOException: <root>/execroot/repo/bazel-out/darwin_arm64-fastbuild/bin/exec_output.runfiles/repo/lib (File exists)
Target //:output failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.105s, Critical Path: 0.01s
INFO: 4 processes: 4 internal.
FAILED: Build did NOT complete successfully

From debugging, it seems that the FileSystemUtils.ensureSymbolicLink function does not do enough to ensure a symbolic link is created (i.e. if there is a folder where the symlink is supposed to go). This happens in the repro1.sh case.

In the second case repro2.sh it seems like a symlink is followed where it should not be:

$ ./repro2.sh
[...]
ERROR: /private/tmp/repr/BUILD.bazel:21:7: Creating runfiles tree bazel-out/darwin_arm64-fastbuild/bin/exec_output.runfiles failed: java.io.IOException:<root>/execroot/repo/bazel-out/darwin_arm64-fastbuild/bin/exec_output.runfiles/repo/lib/sample1.txt (File exists)
Target //:output failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.088s, Critical Path: 0.00s
INFO: 3 processes: 3 internal.
FAILED: Build did NOT complete successfully

The file that the SymlinkTreeAction is trying to overwrite is actually outside the runfiles root:

$ realpath <root>/execroot/repo/bazel-out/darwin_arm64-fastbuild/bin/exec_output.runfiles/repo/lib/sample1.txt
<root>/execroot/repo/bazel-out/darwin_arm64-fastbuild/bin/lib/sample1.txt

In practice we have bumped into this by when migrating from rules_node to rules_js and when trying to optionally pull out some pre-built artifacts.

Which category does this issue belong to?

Core

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I have attached a zip file with a minimal reproduction example, there are two errors that can be produced by running repro1.sh or the repro2.sh scripts.

Which operating system are you running Bazel on?

MacOS

What is the output of bazel info release?

release 6.4.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

I have tested with bazel 6.3.0, 6.4.0 and latest master (sha 487bb42410488a3919e2b4d0c7fd2e37efe8e830), this is reproducible on all the tested versions

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[carpenterjc]

This sounds like its a similar area to: #19018 where the symlink action can happen before the action that creates the linked target is executed.

[tjgq]

Note that this issue is specifically about --experimental_inprocess_symlink_creation, which is not the default. (In addition, in-process just changes the mechanism through which symlinks are created, it doesn't change when they're created relative to other parts of the build).

[iancha1992]

A fix for this issue has been included in Bazel 7.2.0 RC1. Please test out the release candidate and report any issues as soon as possible.
If you're using Bazelisk, you can point to the latest RC by setting USE_BAZEL_VERSION=7.2.0rc1. Thanks!

[Andrius-B]

I tried 7.2.0rc1 and the issue that we were experiencing is indeed fixed there. Thank you!