Dynamic dependencies from build scripts are missed

[prestist]

(Original issue below; this text written by @cgwalters )

Today some crates like rustix end up dynamically enabling features and dependencies via build.rs, and these are missed by us because we just use cargo metadata which doesn't run build scripts.

---

In the latest release afterburn 5.4.2 we have found that afterburn tarball's errno 0.2.8 seems to be filtered of any real content.
This broke the active release for afterburn 5.4.2 resulting in the dependency vmw_backdoor failing with the following error:

error[E0425]: cannot find function `errno` in crate `errno`
   --> /builddir/build/BUILD/afterburn-5.4.2/vendor/vmw_backdoor/src/backdoor.rs:103:35

The vendor tarball includes two versions of errno, 0.3.0 and 0.2.8, and the 0.3.0 bundle appears to be complete.

...
[[package]]
name = "errno"
version = "0.2.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f639046355ee4f37944e44f60642c6f3a7efa3cf6b78c78a0d989a8ce6c396a1"
dependencies = [
 "errno-dragonfly",
 "libc",
 "winapi",
]

[[package]]
name = "errno"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "50d6a0976c999d473fe89ad888d5a284e55366d9dc9038b1ba2aa15128c4afa0"
dependencies = [
 "errno-dragonfly",
 "libc",
 "windows-sys 0.45.0",
]
...

Cargo vender filter does return all the required contents in errno 0.3.0.

[bgilbert]

coreos/afterburn#934 appears to work around this issue.

[cgwalters]

I'm still digging into this a bit, but it's not looking like a bug in vendor-filterer. Before that PR, afterburn was just specifying x86_64-unknown-linux-gnu.

But what I've narrowed things down to is that actually changing it to

diff --git a/Cargo.toml b/Cargo.toml
index fd8c4bf..d1cfa39 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -25,7 +25,7 @@ tag-message = "Afterburn v{{version}}"
 # support for every matching platform, which isn't necessarily available
 # out of the box.  Use this as a stand-in.  Note that architecture-specific
 # crate dependencies (apart from wasm32) are uncommon in the ecosystem.
-platforms = ["x86_64-unknown-linux-gnu"]
+platforms = ["x86_64-unknown-linux-gnu", "s390x-unknown-linux-gnu"]
 all-features = true
 
 [[bin]]

does correctly bring back in errno-0.2.8. What's interesting is that it's somehow s390x specific because adding aarch64-unknown-linux-gnu doesn't!

(The reason then that coreos/afterburn#934 fixes this is because it has the effect of adding all linux, including s390x-unknown-linux-gnu).

[cgwalters]

For reference exactly where and how did the build fail? Seeing the string /builddir/build/BUILD in the logs makes me pretty confident this is a Koji instance (e.g. Fedora, CentOS/RHEL) - is that the case? But on which architecture did the build fail?

[cgwalters]

I didn't fully verify this but I'm pretty sure this is because of
https://github.com/bytecodealliance/rustix/blob/1caf9039ccf136b84bf5e0f18b93cfc23559e3a7/Cargo.toml#L50
which has s390x specific dependencies.

What most likely changed here is that afterburn indirectly picked up a new rustix somewhere along the way.

Anyways, the real fix for this is the tier = "2" change, because today the set of primary Fedora (+derivatives) architectures are all at least Rust tier 2 targets.

[bgilbert]

This is failing on x86_64:

$ cd afterburn
$ git checkout v5.4.2
$ cargo vendor base
$ cargo build --config 'source.crates-io.replace-with="vv"' --config 'source.vv.directory="base"'
[...]
    Finished dev [unoptimized + debuginfo] target(s) in 51.23s
$ cargo vendor-filterer filtered
[...]
Filtered to target platforms: ["x86_64-unknown-linux-gnu"]
Generated: filtered
$ cargo clean
$ cargo build --config 'source.crates-io.replace-with="vv"' --config 'source.vv.directory="filtered"'
[...]
error[E0425]: cannot find function `errno` in crate `errno`
   --> /home/bgilbert/coreos/afterburn/filtered/vmw_backdoor/src/backdoor.rs:103:35
    |
103 |             let err_code = errno::errno();
    |                                   ^^^^^ not found in `errno`

For more information about this error, try `rustc --explain E0425`.
error: could not compile `vmw_backdoor` due to previous error
warning: build failed, waiting for other jobs to finish...
$ uname -mo
x86_64 GNU/Linux

I agree that tier = "2" works around the issue, but platforms = ["x86_64-unknown-linux-gnu"] is still supported syntax, and the latter produces a vendor directory that won't build on x86_64-unknown-linux-gnu.

[cgwalters]

Ah, right OK.

I think the root of the problem here is still rustix; if I add this patch

diff --git a/Cargo.lock b/Cargo.lock
index e08d5f9..c52d855 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -20,6 +20,7 @@ dependencies = [
  "pnet_base",
  "pnet_datalink",
  "reqwest",
+ "rustix 0.37.6",
  "serde",
  "serde-xml-rs",
  "serde_json",
diff --git a/Cargo.toml b/Cargo.toml
index fd8c4bf..8e13222 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -50,6 +50,7 @@ openssl = "0.10.46"
 pnet_base = ">= 0.26, < 0.34"
 pnet_datalink = ">= 0.26, < 0.34"
 reqwest = { version = ">= 0.10, < 0.12", features = [ "blocking" ] }
+rustix = { version = "0.37", features = ["use-libc"] }
 serde =  { version = "1.0", features = [ "derive" ] }
 serde-xml-rs = ">= 0.4, < 0.7"
 serde_json = "1.0"

Then we do build in that scenario.

I think it's that cargo metadata is choosing not to add the use-libc feature, but it's somehow being turned on when we do a real build?

Another twist here is that just for this project, we're getting [email protected] just from tempfile.

$ cargo tree -i [email protected]
rustix v0.37.6
└── tempfile v3.5.0
    └── afterburn v5.4.2 (/var/srv/walters/src/github/coreos/afterburn)
$

Whereas in other projects for which I use vendor-filterer such as rpm-ostree, we explicitly depend on rustix too, and there we actually do this:
https://github.com/coreos/rpm-ostree/blob/ae7366e0de1cee05f58b7be3c26e7761f54a0b48/Cargo.toml#L52

Ahh wait I think I'm understanding the root problem. And yes it is the build script in rustix .

It's all very twisted because historically, rustix went to quite some effort to avoid depending on libc by default on e.g. {x86_64,aarch64}-unknown-linux-gnu.

Then much more recently, bytecodealliance/rustix@a2c6c7a happened and we end up basically depending on libc by default - except this is only enabled by the build script indirectly!

Ultimately the way cargo vendor-filterer works is by just using cargo metadata which does not invoke build scripts today.

In theory, we could, but there's a lot of obvious downsides to that.

Going to leave this open, but it may be a wontfix.