Skip to content

Commit

Permalink
Trim pip 24.3 post
Browse files Browse the repository at this point in the history
  • Loading branch information
@ichard26
ichard26 committed Nov 16, 2024
1 parent f3fdc5d commit 68c9e92
Showing 1 changed file with 20 additions and 126 deletions.
146 changes: 20 additions & 126 deletions content/blog/2024/pip-24.3.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@
title: What's new in pip 24.3
slug: whats-new-in-pip-24.3
description: &desc >-
pip 24.3 is a small release with a truststore bugfix, QoL improvements, and one
pip 24.3 is a small release with a truststore bugfix, error QoL improvements, and
one
minor
deprecation of noncompliant wheel filenames.
summary: *desc
Expand All @@ -11,21 +12,21 @@ tags: [pip, release]
showToc: true
---

On October 27, 2024, the pip team released pip 24.3. This release was a significantly
On October 27, 2024, [the pip team released pip 24.3]. This release was a significantly
smaller release with only a handful of minor fixes and improvements.

This write-up includes the hotfixes included in pip 24.3.1 released in the same day.
This write-up also includes the hotfixes included in pip 24.3.1 released on the same day.

And yes, I know this post is awfully late. It's hard to find time to write these. Better
late then never, however!
Thank you to Damian Shaw, Hugo van Kemenade, and Stéphane Bidoul for reviewing the draft
of this write-up.

## PSA: Legacy editable installs are deprecated

> TL;DR, **don't panic**. The `-e` option is _not_ deprecated, but the way it works under
> the hood will change, potentially necessitating changes to how your project is packaged.
If you aren't aware, since pip 24.2, legacy editable installs are deprecated and **support
is scheduled for removal in pip 25.0 in the new year** (as of writing).
is scheduled for removal in pip 25.0 in the new year** (at the time of writing).

```console {.command hl_lines=[5]}
pip install -e temp/pip-test-package
Expand Down Expand Up @@ -55,7 +56,7 @@ but all you need to know is that:
declares your package's build system to be setuptools**
([see this issue for an example][deprecation])
- **You can keep your `setup.py` file.** It's a configuration file for setuptools, and
setuptools still supports it. Pip simply won't be running it directly anymore,
setuptools still supports it. pip simply won't be running it directly anymore,
delegating to setuptools.
- If you wish, you can migrate your packaging setup to `pyproject.toml` entirely, but
this isn't necessary
Expand All @@ -76,24 +77,10 @@ Imagine you had a requirement file that recursively included itself:
If you ran `pip install -r requirements.txt`, you'd get a nasty crash as pip kept
following the loop until it reached the recursion limit:

<details>
<summary>Show lengthy error</summary>

```console
ERROR: Exception:
Traceback (most recent call last):
File "/home/ichard26/dev/oss/pip/src/pip/_internal/cli/base_command.py", line 105, in _run_wrapper
status = _inner_run()
^^^^^^^^^^^^
File "/home/ichard26/dev/oss/pip/src/pip/_internal/cli/base_command.py", line 96, in _inner_run
return self.run(options, args)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ichard26/dev/oss/pip/src/pip/_internal/cli/req_command.py", line 67, in wrapper
return func(self, options, args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ichard26/dev/oss/pip/src/pip/_internal/commands/install.py", line 343, in run
reqs = self.get_requirements(args, options, finder, session)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[trimmed...]
File "/home/ichard26/dev/oss/pip/src/pip/_internal/cli/req_command.py", line 255, in get_requirements
for parsed_req in parse_requirements(
File "/home/ichard26/dev/oss/pip/src/pip/_internal/req/req_file.py", line 151, in parse_requirements
Expand All @@ -109,127 +96,39 @@ Traceback (most recent call last):
[Previous line repeated 974 more times]
File "/home/ichard26/dev/oss/pip/src/pip/_internal/req/req_file.py", line 337, in _parse_and_recurse
for line in self._parse_file(filename, constraint):
File "/home/ichard26/dev/oss/pip/src/pip/_internal/req/req_file.py", line 374, in _parse_file
args_str, opts = self._line_parser(line)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ichard26/dev/oss/pip/src/pip/_internal/req/req_file.py", line 393, in parse_line
parser = build_parser()
^^^^^^^^^^^^^^
File "/home/ichard26/dev/oss/pip/src/pip/_internal/req/req_file.py", line 439, in build_parser
parser = optparse.OptionParser(add_help_option=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python3.12.4/lib/python3.12/optparse.py", line 1191, in __init__
self.set_usage(usage)
File "/opt/python3.12.4/lib/python3.12/optparse.py", line 1266, in set_usage
self.usage = _("%prog [options]")
^^^^^^^^^^^^^^^^^^^^
File "/opt/python3.12.4/lib/python3.12/gettext.py", line 617, in gettext
return dgettext(_current_domain, message)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python3.12.4/lib/python3.12/gettext.py", line 580, in dgettext
t = translation(domain, _localedirs.get(domain, None))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python3.12.4/lib/python3.12/gettext.py", line 523, in translation
mofiles = find(domain, localedir, languages, all=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python3.12.4/lib/python3.12/gettext.py", line 486, in find
val = os.environ.get(envar)
^^^^^^^^^^^^^^^^^^^^^
File "<frozen _collections_abc>", line 807, in get
File "<frozen os>", line 711, in __getitem__
[trimmed...]
RecursionError: maximum recursion depth exceeded
```

</details>

This example is admittedly a bit contrived, but it can happen if you're referencing
requirement files in a chain and forget what you've already included. Now, pip will detect
this loop and raise a proper error.

```
```console {.command }
pip install -r test.txt
ERROR: /home/ichard26/dev/oss/pip/test.txt recursively references itself in test.txt
```

This was contributed by Kuntal Majumder (@hellozee) in [PR #12877]. ✨

_(There was a bug where the error would be raised too often, even when no actual recursion
was present. This was promptly addressed in pip 24.3.1.)_
_(There was a [bug where the error would be raised too often][#13046], even when no actual
recursion was present. This was promptly addressed in pip 24.3.1.)_

### Installed packages with broken dependency metadata now raise a proper error

Since pip 24.1,
[non standard-compliant versions or dependency specifiers are unsupported][legacy-versions].
While pip will ignore packages with this legacy invalid metadata when possible, in other
cases, the offending package had to be uninstall to allow pip to function.
cases, the offending package had to be uninstalled to allow pip to function.

Unfortunately, similar to the issue above, pip would just crash and leave the user a
horrible traceback in these cases.

<details>
<summary> Show another lengthy error </summary>

```console { .command }
pip install celery
Requirement already satisfied: celery in ./venv/lib/python3.12/site-packages (4.4.7)
ERROR: Exception:
Traceback (most recent call last):
File "/home/ichard26/dev/oss/pip/src/pip/_vendor/packaging/requirements.py", line 36, in __init__
parsed = _parse_requirement(requirement_string)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ichard26/dev/oss/pip/src/pip/_vendor/packaging/_parser.py", line 62, in parse_requirement
return _parse_requirement(Tokenizer(source, rules=DEFAULT_RULES))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ichard26/dev/oss/pip/src/pip/_vendor/packaging/_parser.py", line 80, in _parse_requirement
url, specifier, marker = _parse_requirement_details(tokenizer)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ichard26/dev/oss/pip/src/pip/_vendor/packaging/_parser.py", line 118, in _parse_requirement_details
specifier = _parse_specifier(tokenizer)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ichard26/dev/oss/pip/src/pip/_vendor/packaging/_parser.py", line 208, in _parse_specifier
with tokenizer.enclosing_tokens(
File "/opt/python3.12.4/lib/python3.12/contextlib.py", line 144, in __exit__
next(self.gen)
File "/home/ichard26/dev/oss/pip/src/pip/_vendor/packaging/_tokenizer.py", line 189, in enclosing_tokens
self.raise_syntax_error(
File "/home/ichard26/dev/oss/pip/src/pip/_vendor/packaging/_tokenizer.py", line 167, in raise_syntax_error
raise ParserSyntaxError(
pip._vendor.packaging._tokenizer.ParserSyntaxError: Expected matching RIGHT_PARENTHESIS for LEFT_PARENTHESIS, after version specifier
pytz (>dev)
~^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/ichard26/dev/oss/pip/src/pip/_internal/cli/base_command.py", line 105, in _run_wrapper
status = _inner_run()
^^^^^^^^^^^^
File "/home/ichard26/dev/oss/pip/src/pip/_internal/cli/base_command.py", line 96, in _inner_run
return self.run(options, args)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ichard26/dev/oss/pip/src/pip/_internal/cli/req_command.py", line 67, in wrapper
return func(self, options, args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ichard26/dev/oss/pip/src/pip/_internal/commands/install.py", line 379, in run
requirement_set = resolver.resolve(
^^^^^^^^^^^^^^^^^
File "/home/ichard26/dev/oss/pip/src/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
result = self._result = resolver.resolve(
^^^^^^^^^^^^^^^^^
File "/home/ichard26/dev/oss/pip/src/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ichard26/dev/oss/pip/src/pip/_vendor/resolvelib/resolvers.py", line 427, in resolve
failure_causes = self._attempt_to_pin_criterion(name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ichard26/dev/oss/pip/src/pip/_vendor/resolvelib/resolvers.py", line 239, in _attempt_to_pin_criterion
criteria = self._get_updated_criteria(candidate)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ichard26/dev/oss/pip/src/pip/_vendor/resolvelib/resolvers.py", line 229, in _get_updated_criteria
for requirement in self._p.get_dependencies(candidate=candidate):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ichard26/dev/oss/pip/src/pip/_internal/resolution/resolvelib/provider.py", line 247, in get_dependencies
return [r for r in candidate.iter_dependencies(with_requires) if r is not None]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[trimmed...]
File "/home/ichard26/dev/oss/pip/src/pip/_internal/resolution/resolvelib/candidates.py", line 401, in iter_dependencies
for r in self.dist.iter_dependencies():
File "/home/ichard26/dev/oss/pip/src/pip/_internal/metadata/importlib/_dists.py", line 215, in iter_dependencies
Expand All @@ -245,8 +144,6 @@ pip._vendor.packaging.requirements.InvalidRequirement: Expected matching RIGHT_P
~^
```
</details>
To address this, a new diagnostic error has been added when encountering a broken
pre-installed package.
Expand Down Expand Up @@ -308,7 +205,7 @@ Here, we know three pieces of information immediately:
essentially, `py3-none-any` is code for this package supports “any Python 3 version”,
“any Python ABI”, and “any architecture.”
Unfortunately, some wheels out in the wild are improperly formatted:
Unfortunately, some wheels out in the wild have improper names, such as:
- `translate_toolkit-1.9.0_pinterest3-py27-none-any.whl`
- `j2cli-0.3.0_1-py2-none-any.whl`
Expand All @@ -322,8 +219,7 @@ In other words, `1.2.0_post1` (explicit) is valid, but `1.2.0_1` (implicit) is
not.[^normalization]
For historical reasons, pip has accepted wheels with this error. This is inconsistent and
can lead to parsing ambiguity[^ambiguity], thus starting with pip 24.3, these wheels are
no longer supported.
confusing, thus starting with pip 24.3, these wheels are no longer supported.
Frankly, virtually no packages today have this issue, so you can forget this deprecation
even exists[^not-a-real-problem], but it is a perfect opportunity to learn about the
Expand All @@ -349,15 +245,12 @@ they are not :)
answer is that the version should be normalized when placed in a wheel filename, and
fortunately for us, `1.2.0-1` normalizes to `1.2.0.post1`.
[^ambiguity]: For example, `translate_toolkit-1.9.0_pinterest3-py27-none-any.whl` could be using `_`
as a separator between the version and build tag (although, it is ambiguous as
`pinterest3` is not a valid build tag).
[^not-a-real-problem]: [While reviewing the deprecation PR][#12918], I scraped the index pages for the top
8000 PyPI packages and could only find 4 ancient wheels that have this quirk in their
filename.
[#12918]: https://github.com/pypa/pip/pull/12918
[#13046]: https://github.com/pypa/pip/issues/13046
[compat-tags]: /blog/2024/08/whats-new-in-pip-24.2/#pip-check-just-got-a-bit-stricter
[confusing crashes]: https://github.com/pypa/pip/issues/8438
[data race crash anymore]: https://github.com/pypa/pip/issues/12666
Expand All @@ -366,6 +259,7 @@ they are not :)
[macos-bug]: https://github.com/pypa/pip/issues/12901
[pep 660]: https://peps.python.org/pep-0660/
[pr #12877]: https://github.com/pypa/pip/pull/12877
[the pip team released pip 24.3]: https://discuss.python.org/t/announcement-pip-24-3-release/69350
[this comment]: https://github.com/pypa/pip/issues/11457#issuecomment-2439645125
[truststore bug]: https://github.com/pypa/pip/pull/12918
[version specifier standard]: https://packaging.python.org/en/latest/specifications/version-specifiers/#implicit-post-releases
Expand Down

0 comments on commit 68c9e92

Please sign in to comment.