• pytest-how-to-ignore-missing-tests
  • Overview
  • Getting Started
    • Prerequisites
    • Installation
  • Reproduce
    • Both examples:
    • When file doesn't exists
    • When test function doesn't exist
    • Sample output

pytest-how-to-ignore-missing-tests

This repository is a minimal example demonstrating lack of handling missing/non-existing tests in pytest. It was created as an example for the discussion on pytest-dev GitHub Discussions: How to ignore missing tests?.

Overview

I have an external system to store test suites. If simplified, it is akin to a testlist file — test nodes are resolved and passed into session.config.args before pytest_collect is executed.

Sometimes, tsetlists get out of sync with repo content - i.e. files/test classes/test functions deleted, but remain in the test list. Pytest raises an error whenever a file, function, or class is not found.

1. --continue-on-collection-errors doesn't help here. The error is still raised, and pytest doesn't execute resolved tests.
2. If the suite contains an non-existing file path, pytest starts pytest_collection and just errors out. try...except inside the hook doesn't even get into the except block. pytest_exception_interact also isn't called.
3. If a file exists, but the test function doesn't, pytest fails before reaching pytest_pycollect_makeitem.

Getting Started

Prerequisites

• Tested with Python 3.12.8

Installation

Clone the repository and install the required packages. If a requirements.txt is provided, use it; otherwise, simply install pytest via pip:

git clone https://github.com/sashko1988/pytest-how-to-ignore-missing-tests.git
cd pytest-how-to-ignore-missing-tests
pip install tox

Reproduce

Both examples:

tox -e missing-test-function,missing-file

When file doesn't exists

tox -e missing-file

When test function doesn't exist

tox -e missing-file

Sample output

❯ tox -e missing-test-function,missing-file
missing-test-function: commands[0]> pytest @test-list-with-missing-test-function.txt --continue-on-collection-errors
======================= test session starts =======================
platform darwin -- Python 3.12.8, pytest-8.3.4, pluggy-1.5.0
cachedir: .tox/missing-test-function/.pytest_cache
rootdir: ~/work_stuff/pytest-how-to-ignore-missing-tests
collected 1 item

====================== no tests ran in 0.00s ======================
ERROR: not found: ~/work_stuff/pytest-how-to-ignore-missing-tests/tests/test_example.py::test_two
(no match in any of [<Module test_example.py>])

missing-test-function: exit 4 (0.14 seconds) ~/work_stuff/pytest-how-to-ignore-missing-tests> pytest @test-list-with-missing-test-function.txt --continue-on-collection-errors pid=85450
missing-test-function: FAIL ✖ in 0.16 seconds
missing-file: commands[0]> pytest @test-list-with-missing-file.txt --continue-on-collection-errors
======================= test session starts =======================
platform darwin -- Python 3.12.8, pytest-8.3.4, pluggy-1.5.0
cachedir: .tox/missing-file/.pytest_cache
rootdir: ~/work_stuff/pytest-how-to-ignore-missing-tests
collected 0 items

====================== no tests ran in 0.00s ======================
ERROR: file or directory not found: tests/test_non_existing.py

missing-file: exit 4 (0.13 seconds) ~/work_stuff/pytest-how-to-ignore-missing-tests> pytest @test-list-with-missing-file.txt --continue-on-collection-errors pid=85451
  missing-test-function: FAIL code 4 (0.16=setup[0.03]+cmd[0.14] seconds)
  missing-file: FAIL code 4 (0.13=setup[0.00]+cmd[0.13] seconds)
  evaluation failed :( (0.34 seconds)- [pytest-how-to-ignore-missing-tests](#pytest-how-to-ignore-missing-tests)