Skip to content

Commit

Permalink
Flip flags for coverage support with remote execution
Browse files Browse the repository at this point in the history
With BwoB enabled by default, these flags are already required with a disk cache. They are not easily discovered and make for a bad onboarding experience for coverage.

Also remove the outdated "RE caveats" section from the coverage docs as all issues have been resolved.

RELNOTES: `--experimental_split_coverage_postprocessing` and `--experimental_fetch_all_coverage_outputs` are now enabled by default.
  • Loading branch information
fmeum committed Jan 15, 2025
1 parent d6ee072 commit c3bc9ef
Show file tree
Hide file tree
Showing 7 changed files with 5 additions and 42 deletions.
28 changes: 0 additions & 28 deletions site/en/configure/coverage.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,32 +86,6 @@ To view the result, simply open the `index.html` file produced in the
For further help and information around the `genhtml` tool, or the
`lcov` coverage format, see [the lcov project][lcov].

## Remote execution {:#remote-execution}

Running with remote test execution currently has a few caveats:

- The report combination action cannot yet run remotely. This is
because Bazel does not consider the coverage output files as part of
its graph (see [this issue][remote_report_issue]), and can therefore
not correctly treat them as inputs to the combination action. To
work around this, use `--strategy=CoverageReport=local`.
- Note: It may be necessary to specify something like
`--strategy=CoverageReport=local,remote` instead, if Bazel is set
up to try `local,remote`, due to how Bazel resolves strategies.
- `--remote_download_minimal` and similar flags can also not be used
as a consequence of the former.
- Bazel will currently fail to create coverage information if tests
have been cached previously. To work around this,
`--nocache_test_results` can be set specifically for coverage runs,
although this of course incurs a heavy cost in terms of test times.
- `--experimental_split_coverage_postprocessing` and
`--experimental_fetch_all_coverage_outputs`
- Usually coverage is run as part of the test action, and so by
default, we don't get all coverage back as outputs of the remote
execution by default. These flags override the default and obtain
the coverage data. See [this issue][split_coverage_issue] for more
details.

## Language-specific configuration

### Java
Expand All @@ -127,5 +101,3 @@ for additional steps needed to enable coverage support in Python.

[lcov]: https://github.com/linux-test-project/lcov
[bazel_toolchains]: https://github.com/bazelbuild/bazel-toolchains
[remote_report_issue]: https://github.com/bazelbuild/bazel/issues/4685
[split_coverage_issue]: https://github.com/bazelbuild/bazel/issues/4685
Original file line number Diff line number Diff line change
Expand Up @@ -279,7 +279,7 @@ public static class TestOptions extends FragmentOptions {

@Option(
name = "experimental_fetch_all_coverage_outputs",
defaultValue = "false",
defaultValue = "true",
documentationCategory = OptionDocumentationCategory.UNCATEGORIZED,
effectTags = {OptionEffectTag.AFFECTS_OUTPUTS, OptionEffectTag.LOADING_AND_ANALYSIS},
metadataTags = {OptionMetadataTag.EXPERIMENTAL},
Expand All @@ -301,7 +301,7 @@ public static class TestOptions extends FragmentOptions {

@Option(
name = "experimental_split_coverage_postprocessing",
defaultValue = "false",
defaultValue = "true",
documentationCategory = OptionDocumentationCategory.EXECUTION_STRATEGY,
effectTags = {OptionEffectTag.EXECUTION},
metadataTags = {OptionMetadataTag.EXPERIMENTAL},
Expand Down
3 changes: 1 addition & 2 deletions src/test/shell/bazel/bazel_coverage_cc_test_gcc.sh
Original file line number Diff line number Diff line change
Expand Up @@ -906,7 +906,7 @@ int main() {
return 0;
}
EOF
bazel coverage --test_output=all --experimental_split_coverage_postprocessing //:hello-test \
bazel coverage --test_output=all --experimental_split_coverage_postprocessing --noexperimental_fetch_all_coverage_outputs //:hello-test \
&>$TEST_log && fail "Expected test failure" || :

assert_contains '--experimental_split_coverage_postprocessing depends on --experimental_fetch_all_coverage_outputs being enabled' $TEST_log
Expand All @@ -932,7 +932,6 @@ EOF
bazel coverage --test_output=all \
//empty_cov:empty-cov-test &>"$TEST_log" \
|| fail "Coverage for //empty_cov:empty-cov-test failed"
expect_log "WARNING: There was no coverage found."
}

function setup_external_cc_target() {
Expand Down
2 changes: 1 addition & 1 deletion src/test/shell/bazel/bazel_coverage_sh_test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -505,7 +505,7 @@ exit 0
EOF
chmod +x pull-test.sh

bazel coverage --test_output=all --experimental_fetch_all_coverage_outputs //:pull &>$TEST_log \
bazel coverage --test_output=all --experimental_fetch_all_coverage_outputs --noexperimental_split_coverage_postprocessing //:pull &>$TEST_log \
|| fail "Coverage failed"

local coverage_file_path="$(dirname $( get_coverage_file_path_from_test_log ))/_coverage/foo.txt"
Expand Down
2 changes: 1 addition & 1 deletion src/test/shell/bazel/bazel_execlog_test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -219,7 +219,7 @@ EOF
[[ -e output.compact ]] || fail "no compact log produced"

rm output.compact
bazel coverage //:test --experimental_split_coverage_postprocessing --experimental_fetch_all_coverage_outputs \
bazel coverage //:test --noexperimental_split_coverage_postprocessing --noexperimental_fetch_all_coverage_outputs \
--execution_log_compact_file=output.compact >> $TEST_log 2>&1 || fail "coverage failed"
[[ -e output.compact ]] || fail "no compact log produced"
}
Expand Down
2 changes: 0 additions & 2 deletions src/test/shell/bazel/remote/build_without_the_bytes_test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -1915,8 +1915,6 @@ EOF

bazel coverage \
--test_output=all \
--experimental_fetch_all_coverage_outputs \
--experimental_split_coverage_postprocessing \
--remote_download_minimal \
--combined_report=lcov \
--spawn_strategy=remote \
Expand Down
6 changes: 0 additions & 6 deletions src/test/shell/bazel/remote/remote_execution_test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2178,8 +2178,6 @@ EOF

bazel coverage \
--test_output=all \
--experimental_fetch_all_coverage_outputs \
--experimental_split_coverage_postprocessing \
--spawn_strategy=remote \
--remote_executor=grpc://localhost:${worker_port} \
--instrumentation_filter=//java/factorial \
Expand Down Expand Up @@ -2507,8 +2505,6 @@ EOF

bazel coverage \
--test_output=all \
--experimental_fetch_all_coverage_outputs \
--experimental_split_coverage_postprocessing \
--spawn_strategy=remote \
--remote_executor=grpc://localhost:${worker_port} \
//"$test_dir":hello-test >& $TEST_log \
Expand Down Expand Up @@ -2643,9 +2639,7 @@ EOF
BAZEL_USE_LLVM_NATIVE_COVERAGE=1 GCOV=llvm-profdata BAZEL_LLVM_COV=llvm-cov CC=clang \
bazel coverage \
--test_output=all \
--experimental_fetch_all_coverage_outputs \
--experimental_generate_llvm_lcov \
--experimental_split_coverage_postprocessing \
--spawn_strategy=remote \
--remote_executor=grpc://localhost:${worker_port} \
//"$test_dir":hello-test >& $TEST_log \
Expand Down

0 comments on commit c3bc9ef

Please sign in to comment.