tests: make our test suite and CI faster

[dgedon]

Currently, we have a great set of tests which take a considerable amount of time for PRs. Potentially we want to mark some more tests as "slow" for the hackathon to ensure faster feedback during development. The goal would be to prioritize core functionality tests for PR validation while allowing more time-intensive tests to run separately at a later stage.

[janfb]

Yes, very good point. Before the hackathon, we should

1. further reduce the set of core tests to make the CI faster. We should then make sure to CD on the PR branch once before merging into main

2. find ways to make the test suite more efficient, e.g., by using fixtures for code snippets that are repeated across a test module, e.g., use a pytest fixture that trains NPE on a linear Gaussian task and then re-use this trained posterior across the linear_gaussian_snpe_test.py module where possible?

[janfb]

another idea would be caching test results across CI runs within a feature branch and then executing only the previously failed tests with every new push:

something like:

    - name: Cache pytest results
      uses: actions/cache@v4
      with:
        path: .pytest_cache
        key: ${{ runner.os }}-pytest-${{ github.ref }}
        restore-keys: |
          ${{ runner.os }}-pytest-

    - name: Run only failing tests from last CI run
      run: |
        pytest --lf -v -x -n auto -m "not slow and not gpu" --cov=sbi --cov-report=xml tests/ || \
        pytest -v -x -n auto -m "not slow and not gpu" --cov=sbi --cov-report=xml tests/

note the || between the pytest calls, which will execute all tests only if the any recently failed tests fail again.

NOTE: this seems a bit hacky and should be tested carefully!

[janfb]

yet another idea to speed up CI would be to call only the tests of the changed files, e.g., using https://testmon.org/

this could look something like:

• collect tests affected by the current files changes into .testmondata
• cache this file across CI runs
• run pytest with testmon option

- name: Cache testmon data
  uses: actions/cache@v4
  with:
    path: .testmondata
    key: testmon-${{ github.ref }}-${{ github.sha }}
    restore-keys: |
      testmon-${{ github.ref }}-
      testmon-

- name: Run the fast CPU tests with testmon
  run: |
    pytest --testmon -v -x -n auto -m "not slow and not gpu" --cov=sbi --cov-report=xml tests/