Backwards-incompatible changes - SUBSCRIBE TO THIS THREAD if you use trio!

[njsmith]

Stability is great for users! It lets them focus on solving their problem without worrying about their platform shifting under their feet. But stability is also bad for users! It means that anywhere an API is error-prone or hard to use or missing important functionality, they're stuck with it. Making the optimal trade-off here is tricky and context-dependent.

Trio is a very young project, that contains lots of new ideas, and that doesn't have much code built on top of it yet. So as we build things with it we'll probably discover places where the API is less awesome than desired, and for now we'll be relatively aggressive about fixing them. Hopefully we won't discover any real stinkers, but you never know. Then over time we'll gradually transition over to become more stable as we flush out the bad stuff and get more users.

This means that if you're an early adopter of Trio, it'd be good to have some strategy to make this as painless as possible. Our suggestions:

• Pin your version. For example, in your install_requires= or requirements.txt, do not write: trio >= 0.1.0. Instead, write: trio ~= 0.1.0.

  • You can also use == if you prefer. The difference is that while both == 0.1.0 and ~= 0.1.0 will disallow upgrading to 0.2.0, ~= allows upgrading to 0.1.1 but == 0.1.0 does not. Our intention is that 0.x.y and 0.x.(y+1) will be backwards compatible.

• Please do report back on how trio is working out for you, e.g. by posting a comment on this issue.

  • Especially any rough spots you ran into where the API wasn't as helpful as it could be.
  • Especially if you didn't run into any rough spots, because that information is incredibly valuable in helping us decide when to declare things stable!

• Subscribe to this issue (for example, by pressing the little "Subscribe" button in the right column →). We'll bring up backwards-incompatible changes here before we make them, so this will give you fair warning and a chance to give feedback.

[njsmith]

ahem tap tap is this thing on? Github doesn't give any way to see who's subscribed to an issue, so maybe I'm just talking to myself? Anyway, if anyone is listening, then welcome to the first update on backwards incompatibility in Trio. I think as far as the code goes what's in master right now is pretty much what will be in 0.2.0, but I still want to do some doc updates so it'll still probably be a few days before the actual release. That means that now would be a great time to read this and check out master and let me know if I've made any embarrassing mistakes or am gratuitously inconveniencing anyone.

In general, it turns out that there are very few actual breaking changes in 0.2.0; almost all the changes were easy enough to slap a small layer of back-compat shims on top of. The shims all print noisy TrioDeprecationWarnings that attempt to contain helpful information – suggestions for how to make them more helpful are welcome.

If you read just one section of this comment then read this one

My plan is to be quite aggressive and drop the back-compat shims in 0.3.0. So with a few exceptions (see below), the idea is that:

• if your code works on 0.1.0, it should work on 0.2.0, possibly with warnings
• if you fix the warnings in 0.2.0, then your code should also work on 0.3.0
• jumping directly from 0.1.0 to 0.3.0 will probably break

Important exception for anyone writing network servers: any code using bind (the socket method) is going to break when going from 0.2.0 → 0.3.0, because it needs to become async (#241), BUT there's no way to provide a gentle transition here. Feh. See below for more details and how to mitigate this.

Important exception for anyone using PyPy3: our backcompat shims rely on a feature that's in all supported releases of CPython, and... not yet any released version of PyPy3. (For any time travellers reading this in the future, it's PyPy3 5.8 that's missing the feature, and the upcoming PyPy3 5.9 should have the fix.) So the non-deprecated API is fully supported and should work identically on both CPython and PyPy3, but if you're using deprecated APIs and want to see the pretty warning messages then you need to upgrade to the latest PyPy3 nightly or stick with CPython.

The plan for 0.2.0

Actual breaking changes

In trio.socket, getprotobyname is now async (it does disk I/O), and getservbyport, getservbyname, and getfqdn have been removed (they're buggy and obsolete). These are really obscure APIs, and my guess is that this affects exactly zero people, but if I'm wrong I'd be interested to know.

If sock.sendall is cancelled or raises an error, then it used to attach some metadata to the exception recording how much data it successfully sent. It no longer does this, because it becomes really fragile and error-prone if you have stacked protocol handlers (e.g. if you have an SSLStream wrapped around a SocketStream wrapped around a socket, and SSLStream.send_all raises an exception with bytes-sent metadata attached, is that talked about bytes at the socket layer or at the SSL layer?). Again I'm guessing that no-one was using this, but lmk.

I think that's it. If you try latest master and run into anything else then please let me know ASAP.

Deprecations

Almost all of the nursery API has been deprecated. Most importantly, nursery.spawn is being renamed to nursery.start_soon. The rename of spawn is for consistency with the new nursery.start, see #284, and the deprecation of the rest is to enable "simplified nurseries" with less fiddly requirements around parenting (see #136).

The socket sendall method has been deprecated (#291). Use the new higher-level SocketStream API instead. (See the discussion of bind below for some quick tips.)

The various "run this" APIs have been reorganized to be more consistent and less confusing (see #68). In particular, run now always refers to functions that run an async function, and run_sync is used for functions that run a sync function:

• trio.run_in_worker_thread is now trio.run_sync_in_worker_thread
• trio.current_run_in_trio_thread and trio.current_await_in_trio_thread have been replaced by the trio.BlockingTrioPortal class which has methods run_sync and run
• trio.hazmat.current_call_soon_thread_and_signal_safe has been replaced by trio.hazmat.current_trio_token which returns a trio.hazmat.TrioToken that has a run_sync_soon method.

The following objects have been moved from the main trio namespace into trio.hazmat:

• Task
• UnboundedQueue
• Result
• Error
• Value
• current_task
• current_clock
• current_statistics

In general, "yield points" have been renamed to the less confusing "checkpoints", and functions are being updated to match (#157):

• trio.testing.assert_yields → trio.testing.assert_checkpoints
• trio.testing.assert_no_yields → deprecated without replacement b/c not really useful
• trio.hazmat.yield_briefly → trio.hazmat.checkpoint
• trio.hazmat.yield_briefly_no_cancel → trio.hazmat.cancel_shielded_checkpoint
• trio.hazmat.yield_if_cancelled → trio.hazmat.checkpoint_if_cancelled
• trio.hazmat.yield_indefinitely → trio.hazmat.wait_task_rescheduled

trio.current_instruments is deprecated (#257). Some equivalent will probably return in the future, but in trio.hazmat and returning a read-only snapshot instead of a live object you can mutate.

In trio.Queue, the task_done and join methods have been deprecated; see #321.

The plan (so far) for 0.3.0

As noted, all the changes mentioned in the "Deprecated" section above will become final in 0.3.0. In addition, there's a change coming in 0.3.0 where it's impossible to provide a useful warning period 😞: making the socket bind method async (#241). And unfortunately, in 0.1.0 if you wanted to write a network server at all then you had to call bind.

Fortunately, 0.2.0 adds a new high-level networking API that you'll want to switch to anyway instead of working with raw sockets, and as a bonus, if you do this now then you won't have to worry about 0.3.0's bind and sendall changes.

The quick cheat sheet for switching from the socket API to the stream API is:

• Clients: instead of making a socket and calling connect, use trio.open_tcp_stream.
• Servers: instead of making a socket and calling bind/listen/accept and then spawning off per-connection handlers, use trio.serve_tcp(port, connection_handler).
• Clients and servers: replace sendall with send_all, and recv with receive_some.

That's pretty much it!

Phew

Okay, I think that's everything for 0.2.0. Hopefully this will be the biggest pile of deprecations like this; I wanted to try to clear up as much as possible ASAP before we get more users. Good luck and if you have any feedback please let me know!

[njsmith]

0.3.0 deprecations and breaking changes

I'm just putting together the 0.3.0 release, and it looks like it's pretty simple:

• The changes described above as happening in 0.3.0, are in fact happening. The big thing is bind becoming async, but aside from that if your code runs on 0.2.0 without warnings, then you should be fine.

• I also deprecated the socket methods resolve_local_address and resolve_remote_address. Stdlib socket methods like bind and connect accept either IP addresses or hostnames, and if they get hostnames they implicitly resolve them to IP addresses. Previously in Trio though our socket methods only accepted IP addresses, and if you were porting from stdlib code you'd have to replace sock.connect((name, port)) with addr = await sock.resolve_remote_address((name, port)); await sock.connect(addr). Starting in 0.3.0, though, you can just write await sock.connect((name, port)). See Auto-resolve addresses in socket methods #377. So resolve_{local,remote}_address aren't useful anymore, and have been deprecated.

[njsmith]

Change in policy

When I originally created this issue, we had no deprecation system at all and I had no idea how much churn we'd end up having in our APIs :-). These days we have some pretty solid tooling for documenting and issuing deprecation warnings, and for most of incompatible changes we've had to make it's turned out to be pretty easy to have a transitional warning period. So I'm going to stop posting detailed notes here for every release – you can read the official changelog :-). Instead, I'll save this issue for warning about changes that are more complicated / disruptive / controversial, the kind where it isn't enough to keep up to date with the latest trio release and fix deprecation warnings when you see them.

[njsmith]

Hi all! I just made a post in our forum asking what we should prioritize working on – if you have thoughts, let us know :-)

https://trio.discourse.group/t/priorities-roadmap/34

(By the way, we have a forum now!)

[plugwash]

Hi

A new version of python-msrest in Debian picked up a dependency on python-trio.

Currently python-trio is in Debian unstable, but it is currently blocked from migrating to testing by a bug filed by Robbie Basak saying "Trio upstream do not yet consider the API stable, so in my opinion this package is not yet ready for a stable Debian release". This (along with another issue) is in turn blocking the migration of the new version of python-msrest to Debian testing.

I opened a discussion at https://alioth-lists.debian.net/pipermail/python-modules-team/2019-October/059368.html and also spoke to Robbie Basak in person yesterday and we would like to get your opinion on this. Do you think trio is sufficiently stable to include in the next release of Debian or should Debian disable the part of msrest that depends on trio?

(note that trio has already been included in ubuntu LTS...)

Note that personally I don't use these packages, i'm just looking at this from a Debian QA perspective.

[njsmith]

Do you think trio is sufficiently stable to include in the next release of Debian or should Debian disable the part of msrest that depends on trio?

Well, it depends :-). What does Debian look at when deciding which things are allowed into testing and which are kept in unstable?

We are still being fairly aggressive about fixing API stuff. And by "aggressive" I mean "our deprecation warnings are visible by default and we usually only issue them for a few months before dropping support for the old way", which I guess is much more aggressive than some projects and much less aggressive than others.

OTOH the code we ship is extremely stable in the sense of "it works as advertised" – we have an extremely thorough test suite, high code coverage, etc. So even if Debian got stuck on some Trio release for a while, I don't think you'd need to worry that your Trio package would explode under your feet; it just might get out-of-sync with upstream. But of course "getting out of sync with upstream" is kind of the whole point of Debian's stable releases, so that should be fine?

Also, FWIW, the end goal of all these changes is to get to the point where we can declare the API stable ASAP. So if Debian sticks to the roughly-two-year release cycle they've been on recently then things will probably look very different by the time bullseye freeze rolls around.

[smurfix]

I'm with @njsmith here (and also wearing my DD hat, even if it's kindof rarely used these days): by the time the next Debian release freeze looms Trio should be more than OK for long-term support.

While Trio is not stable-as-in-API-frozen, it's stable as in "very well tested, and perfectly usable long-term in its current state", so yes I'd be happy with allowing it into Testing. If we do deprecate something, any dependents it has know they'll need to follow along before the deprecated parts are removed – but that's already better, frankly, than quite a few supposedly-stable libraries out there.

[basak]

Thank you @njsmith and @smurfix for your comments. Today I've caught up with Trio and dependencies in Debian. Debian unstable is now on 0.13.0, and based on your comments I've closed the blocking bug so we will no longer artificially hold Trio in Debian unstable and allow it to make the next release (assuming it passes the other usual criteria).

[smurfix]

Thank you. Might you be persuaded to also package anyio?

[A5rocks]

I forgot about updating this issue regarding the strict_exceptiongroups=True default change in v0.25.0. See release notes for more information. You can switch it back either globally (with trio.run's parameter) or per nursery (with trio.open_nursery's parameter). See also https://trio.readthedocs.io/en/stable/reference-core.html#designing-for-multiple-errors.

Additionally, I plan on changing the main branch's name for trio from master to main in a few days. Any hardcoded branch names in any scripts involved with trio might need to change; hopefully not.

[A5rocks]

0.27.0 has a breaking change as move_on_after (and fail_after) will now end based on when the context manager is entered rather than based on when they are called. However, this only will make your timeout take longer and would cause too much churn to deprecate, so we did the change without a deprecation period.

[graingert]

A heads up that I've just merged a breaking change, but I hope everyone finds that it's an improvement worth the break: #3110

Rework KeyboardInterrupt protection to track code objects, rather than frames,
as protected or not. The new implementation no longer needs to access
frame.f_locals dictionaries, so it won't artificially extend the lifetime of
local variables. Since KeyboardInterrupt protection is now imposed statically
(when a protected function is defined) rather than each time the function runs,
its previously-noticeable performance overhead should now be near zero.
The lack of a call-time wrapper has some other benefits as well:

• :func:inspect.iscoroutinefunction and the like now give correct answers when
  called on KI-protected functions.

• Calling a synchronous KI-protected function no longer pushes an additional stack
  frame, so tracebacks are clearer.

• A synchronous KI-protected function invoked from C code (such as a weakref
  finalizer) is now guaranteed to start executing; previously there would be a brief
  window in which KeyboardInterrupt could be raised before the protection was
  established.

One minor drawback of the new approach is that multiple instances of the same
closure share a single KeyboardInterrupt protection state (because they share a
single code object). That means that if you apply
@enable_ki_protection <trio.lowlevel.enable_ki_protection> to some of them
and not others, you won't get the protection semantics you asked for. See the
documentation of @enable_ki_protection <trio.lowlevel.enable_ki_protection>
for more details and a workaround.

Here's the additional docs:

Since KeyboardInterrupt protection is tracked per code object, any attempt to
conditionally protect the same block of code in different ways is unlikely to behave
how you expect. If you try to conditionally protect a closure, it will be
unconditionally protected instead::

       def example(protect: bool) -> bool:
           def inner() -> bool:
               return trio.lowlevel.currently_ki_protected()
           if protect:
               inner = trio.lowlevel.enable_ki_protection(inner)
           return inner()

       async def amain():
           assert example(False) == False
           assert example(True) == True  # once protected ...
           assert example(False) == True  # ... always protected

       trio.run(amain)

If you really need conditional protection, you can achieve it by giving each
KI-protected instance of the closure its own code object::

       def example(protect: bool) -> bool:
           def inner() -> bool:
               return trio.lowlevel.currently_ki_protected()
           if protect:
               inner.__code__ = inner.__code__.replace()
               inner = trio.lowlevel.enable_ki_protection(inner)
           return inner()

       async def amain():
           assert example(False) == False
           assert example(True) == True
           assert example(False) == False

       trio.run(amain)

(This isn't done by default because it carries some memory overhead and reduces
the potential for specializing optimizations in recent versions of CPython.)