rules_go 0.40+ broke gopls resolving stdlib imports in VS code

[bluec0re]

What version of rules_go are you using?

Tested with 0.40.1 and 0.42

What version of gazelle are you using?

0.31.1 and 0.33

What version of Bazel are you using?

6.4.0

Does this issue reproduce with the latest releases of all the above?

Yes

What operating system and processor architecture are you using?

Linux amd64

Any other potentially useful information about your toolchain?

Tested with gopls 0.13.2 and 0.14.1.

Using these settings in VS code:

 "go.goroot": "${workspaceFolder}/bazel-${workspaceFolderBasename}/external/go_sdk/",
  "go.toolsEnvVars": {
    "GOPROXY": "off",
    "GOPACKAGESDRIVER": "${workspaceFolder}/scripts/gopackagesdriver.sh",
  },
  "gopls": {
    "build.directoryFilters": [
      "-bazel-bin",
      "-bazel-out",
      "-bazel-testlogs",
      "-bazel-${workspaceFolderBasename}"
    ],
    [...]
  },

and this gopackagesdriver.sh:

#!/bin/sh

bazel="$(command -v bazel || command -v bazelisk)"
# reset GOPROXY to default
unset GOPROXY

# don't fetch internal stuff
export GOPRIVATE="[...]*"

exec "${bazel}" run -- @io_bazel_rules_go//go/tools/gopackagesdriver "$@"

What did you do?

Updated rules_go from 0.39.1 to 0.42 (and then later backtracked the problem to 0.40)

What did you expect to see?

gopls is able to resolve stdlib imports in VS code as before.

What did you see instead?

A lot of import errors in the format of could not import $PKG (cannot find packge "$PKG" in GOROOT or GOPATH).

[JamyDev]

Could you provide the output of echo {} | ./scripts/gopackagesdriver.sh builtin and save it on gist.github.com?

[bluec0re]

I saved the output for 0.39.1 and 0.40.1 and also a diff between them.

To be able to generate the diff, it used echo {} | ./scripts/gopackagesdriver.sh builtin | jq '.Packages |= sort_by(.PkgPath)' | jq '.Roots |= sort' to create the files.

[bluec0re]

Interestingly, it shows no error for unsafe but the gopls logs report "no readable files" when using jump to definition:

[Error - 12:38:35 PM] Request textDocument/definition failed.
  Message: package "@io_bazel_rules_go//stdlib:unsafe" has no readable files
  Code: 0 

[JamyDev]

Do you know what gopls version you're running? the outputs you shared seem normal 🤔

[bluec0re]

Sorry for the year long silence. I think I found the reason:

During the initial load of the workspace, gopackagesdriver does not provide any files for "CompiledGoFiles" for stdlib packages. gopls then drops these imports in https://github.com/golang/tools/blob/8171d94fe98a51fb8471b40726983dd098a3fde6/gopls/internal/cache/load.go#L533-L536 (except for "unsafe"). This then results in gopls reporting that the import could not be found (the error now is "go/packages driver could not load $PKG").

I played around with the request as well (gopls sends sth like {"mode":32287,"env":[/* my env vars */], "build_flags":null,"tests":true,"overlay":{}}), and I was able to reproduce the behavior, but only once per request. Sometimes it triggered again when I modified the request, but again only once.

My test one-liner was:

echo '{"mode":32287,"env":[], "build_flags":null,"tests":true,"overlay":{}}' | ./scripts/gopackagesdriver.sh ./... builtin | jq '.Packages[] | select(.ID == "@@rules_go+//stdlib:os") | .CompiledGoFiles | length'

Which then resulted in

echo '{"mode":32287,"env":[], "build_flags":null,"tests":true,"overlay":{}}' | ./scripts/gopackagesdriver.sh ./... builtin | jq '.Packages[] | select(.ID == "@@rules_go+//stdlib:os") | .CompiledGoFiles | length'
INFO: Invocation ID: 323cf33a-95a8-4d8e-9c83-b948f93e79d1
Computing main repo mapping: 
Loading: 
Loading: 0 packages loaded
Analyzing: target @@rules_go+//go/tools/gopackagesdriver:gopackagesdriver (0 packages loaded, 0 targets configured)
Analyzing: target @@rules_go+//go/tools/gopackagesdriver:gopackagesdriver (0 packages loaded, 0 targets configured)

INFO: Analyzed target @@rules_go+//go/tools/gopackagesdriver:gopackagesdriver (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target @@rules_go+//go/tools/gopackagesdriver:gopackagesdriver up-to-date:
  bazel-bin/external/rules_go+/go/tools/gopackagesdriver/gopackagesdriver_/gopackagesdriver
INFO: Elapsed time: 0.179s, Critical Path: 0.04s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/external/rules_go+/go/tools/gopackagesdriver/gopackagesdriver_/gopackagesdriver <args omitted>
Running: [bazel info --tool_tag=gopackagesdriver --ui_actions_shown=0]
INFO: Invocation ID: 02da1fc2-23a5-45ea-9cfb-a30c50c8a8ef
Running: [bazel query --tool_tag=gopackagesdriver --ui_actions_shown=0 --consistent_labels --ui_event_filters=-info,-stderr --noshow_progress --order_output=no --output=label --nodep_deps --noimplicit_deps --notool_deps kind("^(go_library|go_test) rule$", ...) union @@rules_go+//:stdlib]
Running: [bazel build --tool_tag=gopackagesdriver --ui_actions_shown=0 --show_result=0 --build_event_json_file=/tmp/gopackagesdriver_bep_1686033684 --build_event_json_file_path_conversion=no --experimental_convenience_symlinks=ignore --ui_event_filters=-info,-stderr --noshow_progress --aspects=@@rules_go+//go/tools/gopackagesdriver:aspect.bzl%go_pkg_info_aspect --output_groups=go_pkg_driver_json_file,go_pkg_driver_stdlib_json_file,go_pkg_driver_srcs --keep_going /* ... */ @@rules_go+//:stdlib]
0

but then subsequent runs result in this:

echo '{"mode":32287,"env":[], "build_flags":null,"tests":true,"overlay":{}}' | ./scripts/gopackagesdriver.sh ./... builtin | jq '.Packages[] | select(.ID == "@@rules_go+//stdlib:os") | .CompiledGoFiles | length'
INFO: Invocation ID: 6b89bf73-a78e-4c1e-8daf-7f363f5141ec
Computing main repo mapping: 
Loading: 
Loading: 0 packages loaded
Analyzing: target @@rules_go+//go/tools/gopackagesdriver:gopackagesdriver (0 packages loaded, 0 targets configured)
Analyzing: target @@rules_go+//go/tools/gopackagesdriver:gopackagesdriver (0 packages loaded, 0 targets configured)

INFO: Analyzed target @@rules_go+//go/tools/gopackagesdriver:gopackagesdriver (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target @@rules_go+//go/tools/gopackagesdriver:gopackagesdriver up-to-date:
  bazel-bin/external/rules_go+/go/tools/gopackagesdriver/gopackagesdriver_/gopackagesdriver
INFO: Elapsed time: 0.162s, Critical Path: 0.04s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/external/rules_go+/go/tools/gopackagesdriver/gopackagesdriver_/gopackagesdriver <args omitted>
Running: [bazel info --tool_tag=gopackagesdriver --ui_actions_shown=0]
INFO: Invocation ID: 88749a04-95bb-4159-bc2d-0b131e88ccfd
Running: [bazel query --tool_tag=gopackagesdriver --ui_actions_shown=0 --consistent_labels --ui_event_filters=-info,-stderr --noshow_progress --order_output=no --output=label --nodep_deps --noimplicit_deps --notool_deps kind("^(go_library|go_test) rule$", ...) union @@rules_go+//:stdlib]
Running: [bazel build --tool_tag=gopackagesdriver --ui_actions_shown=0 --show_result=0 --build_event_json_file=/tmp/gopackagesdriver_bep_2296271058 --build_event_json_file_path_conversion=no --experimental_convenience_symlinks=ignore --ui_event_filters=-info,-stderr --noshow_progress --aspects=@@rules_go+//go/tools/gopackagesdriver:aspect.bzl%go_pkg_info_aspect --output_groups=go_pkg_driver_json_file,go_pkg_driver_stdlib_json_file,go_pkg_driver_srcs --keep_going /* ... */ @@rules_go+//:stdlib]
36

[bluec0re]

Correction, it seems to fluctuate:

for i in $(seq 1 10); do echo '{"mode":32287,"env":[], "build_flags":null,"tests":true,"overlay":{}}' | ./scripts/gopackagesdriver.sh ./... builtin 2>/dev/null | jq '.Packages[] | select(.ID == "@@rules_go+//stdlib:os") | .CompiledGoFiles | length'; done
36
0
0
36
0
0
36
0
0
0

[JamyDev]

@bluec0re can you try patching in #4185? I think it may be/solve your issue.

[bluec0re]

That made it worse:

INFO: Invocation ID: b4a32a44-882a-4d8b-a0d8-92e322a72bca
Computing main repo mapping:
Loading:
Loading: 0 packages loaded
Analyzing: target @@rules_go+//go/tools/gopackagesdriver:gopackagesdriver (0 packages loaded, 0 targets configured)
Analyzing: target @@rules_go+//go/tools/gopackagesdriver:gopackagesdriver (0 packages loaded, 0 targets configured)

INFO: Analyzed target @@rules_go+//go/tools/gopackagesdriver:gopackagesdriver (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target @@rules_go+//go/tools/gopackagesdriver:gopackagesdriver up-to-date:
  bazel-bin/external/rules_go+/go/tools/gopackagesdriver/gopackagesdriver_/gopackagesdriver
INFO: Elapsed time: 0.096s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/external/rules_go+/go/tools/gopackagesdriver/gopackagesdriver_/gopackagesdriver <args omitted>
Running: [bazel info --tool_tag=gopackagesdriver --ui_actions_shown=0]
INFO: Invocation ID: 3970c0c1-f492-4836-bc98-5ec1fe3fd7e0
Running: [bazel query --tool_tag=gopackagesdriver --ui_actions_shown=0 --consistent_labels --ui_event_filters=-info,-stderr --noshow_progress --order_output=no --output=label --nodep_deps --noimplicit_deps --notool_deps kind("^(go_library|go_test) rule$", non_pure:*) union @@rules_go+//:stdlib]
Running: [bazel build --tool_tag=gopackagesdriver --ui_actions_shown=0 --show_result=0 --build_event_json_file=/tmp/gopackagesdriver_bep_819825832 --build_event_json_file_path_conversion=no --experimental_convenience_symlinks=ignore --ui_event_filters=-info,-stderr --noshow_progress --aspects=@@rules_go+//go/tools/gopackagesdriver:aspect.bzl%go_pkg_info_aspect --output_groups=go_pkg_driver_json_file,go_pkg_driver_stdlib_json_file,go_pkg_driver_srcs --keep_going @@//non_pure:main_lib @@//non_pure:main_test @@rules_go+//:stdlib]
error: unable to load JSON files: unable to resolve imports: open /home/user/.cache/bazel/_bazel_user/90576f5ebdec5c420fc71497b1c555d5/execroot/_main/bazel-out/k8-fastbuild/bin/external/rules_go+/stdlib_/gocache/e9/e9c8ad96a519cb209153dcc7c08b52145ed768206bdd112fbd0ef94d45697162-d: no such file or directory

But I was able to reduce the problem to a single rule:

Looks like this only happens for packages which have tests with pure = "on" settings. I uploaded a test project here: https://github.com/bluec0re/rules_go_stdlib_missing_files

When running gopackagesdriver for the non_pure package, all works fine. But for the pure one, stdlib packages won't have CompiledGoFiles set, which then lets gopls believe something is wrong.