Not possible to use bottleneck as a dependency on linux arm64 with rules_python 0.37.0 #2328
Comments
|
It looks like it is working as intended to be honest. If the |
|
Could you help me understand a bit more about why this works in rules_python 0.34.0 but no longer works in later versions? I've been tracking the various wheel arg changes but tbh I skipped over understanding the more fundamental changes shipped in 0.35.0 with whl_library builds. Separately, I'm at a bit of a loss for how to move forward -- it seems like rules_python will be generally incompatible with any sdists that have dependencies outside of the blessed set of build deps that rules_python normally brings in, which leaves the options as something like:
Something here will probably work for me, for now, but IMO this is really raising the bar for adopting Bazel for Python. |
|
Alternatively, I'm able to produce a wheel successfully if instead of running |
|
If a C extension is being created, then the standard advice is that the project should be providing a wheel. Building C extension from source can be quite particular to projects. Otherwise, I think the other options are:
|
I'm actually interested in how it was working at all in the first-place myself. It does indicate some sort of non-hermetic scenario in previous versions which have now been made to be hermetic... |
What you can do in this instance @mark-thm , depending on your specific environment and supply chain requirements. Is to upload the wheel to a github release and use direct_url references. So something like: (I just guessed the wheel tags in the example. They might be different based on your specific ABI etc) |
|
The python ecosystem is looking to partially solve some of this in the lockfile standard, but it gets quite gnarly. Different sdist might have different build-time dependencies for example. So you might need to be provided multiple independent build environments. And then some build environments take dependencies on the exec platform if they link out to OS libraries etc. So it becomes quite the rabbithole at the limit. The good news is that the vast majority of major python dependencies are already wheels: https://pythonwheels.com/ And that tools like https://cibuildwheel.pypa.io/en/stable/ exist to help project authors package wheels for most popular platforms. sdist are inevitable though, but the ones that aren't pure-python are becoming more and more rare in my experience. |
|
bottleneck already uses cibuildwheel, but hasn’t set up emulation to publish arm64 wheels because the CI system doesn’t run on arm64 (because GitHub doesn’t yet provide OSS Linux/arm64 runners). I’ve made a PR to add wheel builds but the maintainers (very reasonably) want to also run tests. That’s a bit trickier, I’m not sure there’s a great way to do this for low maintenance cost and for free. I think what feels tricky about rules_python’s current behavior here is a couple of things:
Thus the state of the world is that nobody has done anything wrong (you’re saying expected behavior, dependency maintainer has correctly encoded build time dependencies, pip wheel is behaving as expected), and yet my build is failing. But I do think there’s a broken assumption here: pip wheel demands that you parse the pyproject.toml file and supply dependencies, and that’s not what’s happening here. pip install is a viable alternative that will fetch and build missing dependencies for the build, but that wont use the Bazel downloader (and it seems like rules_python is trying to take control of that interaction). Separately: yes, I can build all my deps into wheels and publish these things somewhere—my normal build orchestration tool is Bazel, so the task ahead is to make something like what rules_python does: parse a requirements files, download sdists, build those sdists using pip, and then also publish those files somewhere. A couple of ideas:
|
|
Yes, that’s a good summary on the issue and possible solutions. We’ve had many similar discussions on issues and as maintainers.
It’s also a reflection of the current python packaging ecosystem realities. Even outside bazel the community struggles with these similar issues. Your upstream PR demonstrates that it isn’t even clear if upstream bottleneck is supported on arm64 by the maintainers. It’s nobody’s fault, rather the tragedy of the commons. The python packaging community has considered disabling sdist by default as a mitigation. Thats the origin of the message “this is likely not a problem with pip”. A similar framing could be taken “this is likely not a problem with bazel/rules_python” (But I’m still really curious how this issue only occurred upon update of the rules. I’m not sure we got to the bottom of what changed?) Emerging lockfile standards, wheel format improvements, index metadata, disabling sdists and installers like uv are all improving. rules_python is downstream of all these things and is also improving. These things just take time.
Are we sure that “pip wheel” outside bazel fails too? I don’t think it will. If I had an arm64 machine I could test it. My understanding of the PEP 517 build hooks is that they are invoked for both pip install and pip wheel. The problem here boils down essentially to: “running other people’s build makefiles”. So it’s a bit unrealistic to solve completely in the presence of non-hermetic, Turing complete systems. Stating that “pip install works” is the same as saying “works on my machine”. I think as you’ve mentioned, the workarounds in order of effort for you are probably:
|
It fails for me, yes: replicating the
I’m not an expert here, but I disagree with this. In particular, I’m trying to call attention to the differences in behavior of pip wheel and pip install. What I observe:
|
|
That’s a misunderstanding I believe. I’ll try a repro on a non arm64 machine to demonstrate. Both subcommands invoke build backends hooks. eg. They both offer “—no-build-isolation” arguments. Indeed, before pip installs any sdist dependency, a wheel is always built first. |
|
Looking at the full wheel command executed in the trace, I think I know why pep517 build hooks aren’t pulling down the build dependencies declared in the bottleneck pyproject.toml. The “—no-build-isolation” option is provided to the wheel command. Removing that will probably fix this sdist, but might break others. |
|
Ok, can confirm that removal of |
Not necessarily. The majority of actively maintained projects should be using PEP-517 metadata to fully specify and configure their build dependencies ( However, there are also a lot of builds that aren't well-behaved. That is why build frontends like I've had a think and I think a good path forward to avoid disappointment, and to make expectations clear to users of the rules who have sdist in their dependency closures, might be to do the following:
So in this Named build environment: Now, that might all be some future implementation. For now, I don't see a strong reason for not removing |
|
Yeah, removing Regarding the named build envs, it is a nice sketch and could solve majority of the issues that we are facing these days. |
|
Pushed #2360 |
Fixes #2328. In #2126, we discussed and added `--no-build-isolation` to ensure that sdist builds pulled from rules_python supplied dependencies. An unanticipated/misunderstood side-effect of this is that pip will no longer resolve necessary, declared build time dependencies from a package index. That's a significant point of friction for users, and, it's much harder to opt-out of the behavior than to opt-in given the presence of `extra_pip_args`.
🐞 bug report
Affected Rule
whl_library
Is this a regression?
Yes, last functioned in rules_python 0.34.0.
Description
It's not possible to use bottleneck as a dependency on linux arm64 with rules_python 0.37.0
🔬 Minimal Reproduction
https://github.com/theoremlp/rules_python_repro
🔥 Exception or Error
🌍 Your Environment
Operating System:
Output of
bazel version:Rules_python version:
Anything else relevant?
Related:
--no-indexand add--no-build-isolationwhen build sdist #2126The text was updated successfully, but these errors were encountered: