Looking for tests to validate runtime type checking

[Josverl]

We are working to add 'just enough' runtime support to MicroPython to support the use of code using inline type annotations.
As MicroPython is intended to run on CPU and memory constrained devices, the goal is to keep the code and runtime overhead as small as possible.
The MicroPython parser already ignores most method and parameter type annotations so the plan is to add very minimal typing, __futures__ , typing_extensions, abc and collection.abc modules, with the primary intent to not fail imports and runtime expressions.
that still leaves a number of expressions/methods that are/need to be evaluated at runtime.
We have a few working prototypes 1,2,3 that appear to work, that I want to compare/validate for correct runtime behavior.

I have gathered samples from the typing docs and https://typing.readthedocs.io/ into a notebook running the code snippets on a MCU or unix version of MicroPython for validation and experimentation.
After we have a solid set of test these will be integrated with the CI/CD .

Part of the challenge is finding what runtime behavior should be tested , and what code snippets to use for that, but I'm pretty sure I have not thought of everything needed.
I have looked for (python based) tests to verify correct runtime behavior,
or even for a list of typing-related runtime behaviors that need testing , and not been able to find that thus far.

Any pointers or assistance is welcome.

• @stinos , @andrewleech

[JelleZijlstra]

There's multiple levels you could go here. At one extreme, if you want to support thorough runtime typing libraries (e.g., Pydantic, typeguard, beartype), you need to basically replicate all of typing.py; these libraries rely heavily on the way CPython represents types.

At the other extreme, many users who use only static typing don't look at the annotations at runtime at all. However, you still need to support those pieces of typing syntax that are not placed in annotations at runtime, such as TypedDict and NamedTuple class definitions, TypeVar and NewType definitions, cast() calls, generic base classes, various decorators, and a few others.

In the middle, there is the light introspection used by some parts of the standard library. For example, @dataclass looks for ClassVar annotations and treats them specially, and functools.singledispatch introspects union types in annotations. I don't know if it's a goal for you to support these standard library features.

[andrewleech]

@JelleZijlstra there's a lot of backstory in the linked discussions, but the basic concept is for development to use cpython and associated type checking tooling in conjunction with micropython stubs (which @Josverl already builds publishes).

To be able to use some desktop type checking tooling and support other libraries that also use desktop type checking tooling micropython itself need to support / ignore a certain level of type annotation / typing features in the python code itself.

MicroPython doesn't need to run the tooling or understand the typing, the syntax just needs to not get in the way or break micropython.

What we need to figure out is just how much of the typing syntax and more "advanced" in-code features are worth supporting (ignoring)

[hauntsaninja]

I think Jelle's answer already has basically everything you need.

Decide whether you want to keep annotations accessible at runtime at all (__annotations__ attribute)

If the answer is yes, you'll likely want to match CPython fairly closely.

If the answer is no, you'll want to make sure the remaining things Jelle mentions mostly work:

TypedDict and NamedTuple class definitions, TypeVar and NewType definitions, cast() calls, generic base classes, various decorators, and a few others

One other decision is whether you care about matching errors, if not, you could maybe even literally use unittest.MagicMock equivalent to stub out the things in typing.py

[Josverl]

Thanks Jelle, Shantuna,

I want to start by aiming low, and if we can find support for popular constructs such as @dataclass that would be welcome. For the foreseeable future not consider support for Pydantic and friends.
There are no current plans to make annotations accessible via __annotations__, there is currently also no __docs__

There are a few thing that I know currently fail at runtime with the solutions due to constraints ( Generics, metaclass , @runtime_checkable, and new Python 3.12 syntax ), and that's fine as long we can document and explain them.

Jelle's list is a good addition , and I have already covered ~70% of that through my own explorations. But Happy to have a few more to add to the test list.
if you have any more details on the : various decorators, and a few others  that would be appreciated.

[hauntsaninja]

There are lots of decorators in typing.py (final, override, no_type_check, ...), these obviously all will do a thing at runtime but should be trivial to stub.

Not sure what's missing from Jelle's list. Maybe isinstance on types.UnionType (i.e. isinstance(x, X | Y)). Subclassing from the Protocols or Any. from __future__ import annotations should avoid errors on forward references. Generally matching errors (e.g. runtime will complain if you subscript a TypeVar), if you care about that at all. But I think you'll just have to iterate and see.

You could see if the tests in https://github.com/python/typing/tree/main/conformance/tests have anything useful for you. Those tests are mainly for static type checkers, not Python runtime implementations, but maybe they'll have some constructs that do something at runtime.

[Josverl]

The tests in the conformance folder are useful, already found a few other gotcha's.