hatch swallows test failures ?

[Carreau]

It seem that hatch (or something else?) jus ignore some failure.

see this run of current main branch:

https://github.com/ipython/ipykernel/actions/runs/13286863326/job/37097508006 which is aborted,
and this comment:

#1322 (comment)

That point to a run with actual failures that appear as passing.

[davidbrochart]

It's because we rerun last failed tests here.

[Carreau]

That seems wrong, it can hide test that modify global state and make subsequent test to fail.
Plus the abort I linked to does not seem to be reran.

[davidbrochart]

Plus the abort I linked to does not seem to be reran.

You mean this? https://github.com/ipython/ipykernel/actions/runs/13286863326/job/37097508006
Yes it is rerun (look for --lf).

[Carreau]

No, your confusing the two run because hatch run tests twice .qt = ["qt5", "qt6"]

The first run abort at 14%, Fatal Python error: Aborted, pytest does not write it's json file with failing test (there are no failed test, and the ones after test_qt_enable_gui are not run, so the second --lf does nothing.

Compare with https://github.com/ipython/ipykernel/actions/runs/13324822814/job/37215765326?pr=1324, where you get two complete full runs.

[Carreau]

Or a more explicit question/prof to show you that test are not properly ran:

Show me where tests/test_heartbeat.py::test_port_bind_failure_raises is ran under cov.qt5 ?,

• It is ran neither by the hatch first invocation (it would be ran after the Abort),
• It is not ran under --lf that reruns nothing.

Same for tests/test_eventloop.py::test_cocoa_loop, which is the crashing test on cov.qt5:

• where is it reran ?

[davidbrochart]

Show me where tests/test_heartbeat.py::test_port_bind_failure_raises is ran under cov.qt5 ?

Here?

[Carreau]

Here?

And no, again this is not under cov.qt5, it's under test, qt is not available !

I also meant q6 insted of 5 as well but it does not change much

None of the qt6 test are run.

This is current report on main branch:

2025-02-12T13:48:26.3781030Z ##[group]Run hatch run cov:test --cov-fail-under 50 || hatch run test:test --lf


2025-02-12T13:48:26.3918128Z + hatch run cov:test --cov-fail-under 50
2025-02-12T13:48:27.3793726Z ─────────────────────────────────── cov.qt5 ────────────────────────────────────
2025-02-12T13:48:27.3796015Z Creating environment: cov.qt5

...  
2025-02-12T13:48:57.2991349Z tests/test_async.py::test_async_await �[32mPASSED�[0m�[33m                             [  0%]�[0m
...
2025-02-12T13:49:34.6306217Z tests/test_eventloop.py::test_enable_gui �[32mPASSED�[0m�[33m                          [ 13%]�[0m
2025-02-12T13:49:34.6322729Z tests/test_eventloop.py::test_cocoa_loop �[33mSKIPPED�[0m (MacOS-only)�[33m            [ 14%]�[0m
2025-02-12T13:49:36.3876305Z Fatal Python error: Aborted
...

^ Here we aborted, all of the test after this won't be marked as failed, all of the qt6 test will not run.

2025-02-12T13:49:38.2130862Z + hatch run test:test --lf

## rerun all the test under Creating environment: test
## You can see the pyqt test are skipped :


2025-02-12T13:50:15.2029784Z tests/test_eventloop.py::test_qt_enable_gui �[33mSKIPPED�[0m (No viable version of PyQt or PySide installed.)�[33m                                            [ 14%]�[0m

^ See SKIPPED No viable version of PyQt or PySide installed

But

If you remove the --lf the failure is not hidden ! This is an actual failure, and then you mark the aborting test as skipping you run test that otherwise are not ran:

In PR #1324:

2025-02-14T07:49:44.9124248Z ─────────────────────────────────── cov.qt5 ────────────────────────────────────
2025-02-14T07:49:44.9126900Z Creating environment: cov.qt5

...Ok same as main with no failure I'm not going to rewrite this but this is with qt5
So everything after tests/test_eventloop.py::test_enable_gui is actually passing with qt5

All of QT6 is is missing from main as well but present in #1324

2025-02-14T07:52:19.1069595Z ─────────────────────────────────── cov.qt6 ────────────────────────────────────
2025-02-14T07:52:19.1071179Z Creating environment: cov.qt6

^ This is not created or ran on main !

...
2025-02-14T07:53:22.0112802Z tests/test_eventloop.py::test_asyncio_loop �[32mPASSED�[0m�[33m                        [ 13%]�[0m
2025-02-14T07:53:23.7892640Z tests/test_eventloop.py::test_enable_gui �[32mPASSED�[0m�[33m                          [ 14%]�[0m

^ This is not ran with qt6 !

Again I'm asking is it reran under qt5 or qt6 not is it reran under test !

The --lf line is

hatch run cov:test --cov-fail-under 50 || hatch run test:test --lf

not

hatch run cov:test --cov-fail-under 50 || hatch run cov:test --lf

and the test env does not have qt, only the cov env has:

# pyproject.toml
[tool.hatch.envs.test]
features = ["test"]
[tool.hatch.envs.cov]
features = ["test", "cov"]

...

[[tool.hatch.envs.cov.matrix]]
qt = ["qt5", "qt6"]

[tool.hatch.envs.cov.overrides]
matrix.qt.features = [
  { value = "pyqt5", if = ["qt5"] },
  { value = "pyside6", if = ["qt6"] },
]

Now you can argue, , yes but for most tests whether qt is installed or not does not matter, well maybe, but for some it does, for example tests/test_eventloop.py::test_enable_gui, and tests/test_eventloop.py::test_enable_gui failure under qt5 is swallowed and nothing get ran under qt6 because tests/test_eventloop.py::test_enable_gui['qt5'] crashes, and because of || --lf

[Carreau]

One more argument that ||--lf is swallowing errors

In #1234 I actually marked some qt5 test as @skip as they were segfaulting, and now we see previously unknown failure on python39 and qt6 !