Timeout not working as expected

[catbrained]

So I have an iterator that calculates prime numbers. I tried running cargo-mutants and saw that some mutations cause timeouts.
Makes sense. If you switch some operator somewhere the algorithm will run forever without finding any primes.

So I wrote a test that is supposed to catch infinite loops:

    #[test]
    fn primes_generates_1000_in_less_than_2s() {
        let (tx, rx) = std::sync::mpsc::channel();
        let _ = std::thread::spawn(move || {
            let _: Vec<_> = Primes::with_capacity(1000).take(1000).collect();
            let _ = tx.send(());
        });

        assert!(rx.recv_timeout(std::time::Duration::from_secs(2)).is_ok());
    }

I ran cargo-mutants again, but it still reports timeouts for these mutations.

Am I misunderstanding something about how cargo-mutants works? Is my test not actually able to catch timeouts for some reason?

EDIT:
I manually changed the code under test to match the timeout mutation that cargo-mutants indicated.
Running cargo test hangs until I Ctrl-C it, while cargo nextest run reports that the timeout test failed, but then it also hangs, reporting that some other test is still running (because it also ran into the timeout issue but doesn't have a check to fail after a certain time).

So it looks like there's something about the way Rust test harnesses work that I don't fully understand.

• Why does cargo-mutants report timeouts even when I run it with nextest as the test runner?
  Is it because the other tests, which do not check for infinite loops, are stuck?
• Can I change my test in some way to fix this? Preferably without copying the timeout logic into every single test.

[sourcefrog]

Yes, you probably have some other test (or code called from a test) that hangs when this mutation is inserted. If you look at the corresponding log inside mutants.out/logs it might be more clear which one it is -- or, as you already did, just insert the same change by hand.

I'm not sure what's happening with nextest reporting a timeout but not stopping. If you can make and share a small repo that shows the behavior then perhaps nextest devs would like to look too.

Can I change my test in some way to fix this? Preferably without copying the timeout logic into every single test.

I'm not sure. One pragmatic thing you can do is just exclude this mutant (or the function containing it), but that's not entirely satisfactory. Another option is to just accept that this mutant causes a timeout, although that does make the mutants run longer and makes the results somewhat inconclusive.

Possibly another option is to insert debug_assert calls within the code that hangs, asserting that it has made some progress and causing the program to immediately fail if it has not. For example in some recursions or loops the working set should get smaller on each iteration. In some programs such assertions might look contrived or over-fitted to the mutants, but perhaps in some programs they would be a good expression of intent or they might guard against naturally occurring bugs?

I'm still looking for better answers here.

[catbrained]

Thank you for the quick reply!

I'm not sure what's happening with nextest reporting a timeout but not stopping. If you can make and share a small repo that shows the behavior then perhaps nextest devs would like to look too.

It turns out I didn't have nextest configured correctly. I changed the timeout settings to slow-timeout = { period = "20s", terminate-after = 2 }, and now it correctly terminates after 40 seconds.

With regards to actually catching infinite loops in tests, I tried your suggestion of adding some debug_assert calls. I did indeed find an invariant to check, and now those mutants are caught. This has the added benefit of failing fast, rather than relying on timeouts!

In any case, it would certainly be nice if cargo-mutants could consider a mutant as caught as soon as any test fails.
That would allow users to write dedicated tests with timeouts which check for infinite loops, for those cases where it might be difficult/contrived to check progress inside the code under test.
As soon as the timeout test fails, the mutant can be marked as caught and all the other tests cancelled.

I found the following issue, which seems to be related: #423. If I understand correctly, adding a feature/improvement like that would be a larger effort though.

Generally speaking, I suspect that a lot of code that might be susceptible to infinite loops could be made easier to test with a few simple debug_asserts without getting too contrived. I don't have any data to back that up though.

Either way, cargo-mutants is already helping me a lot to improve the way I test and think about testing!

[sourcefrog]

In any case, it would certainly be nice if cargo-mutants could consider a mutant as caught as soon as any test fails.

It would be good! The challenge is that the default Rust test runner does not have an option to fail fast within a single test target. (Nextest can do this.)

There is some discussion in

• Notice test failures that accompany hangs #423
• Run one, or a few, tests per subprocess #194

Basically I can think of a few options:

1. Run a few tests at a time, like a mini version of Nextest
2. Or, look at stdout and interrupt the process when it looks like something has failed, much as you might do manually
3. Or, just wait and maybe Rust will fix this upstream (or try to fix it upstream)

That would allow users to write dedicated tests with timeouts which check for infinite loops, for those cases where it might be difficult/contrived to check progress inside the code under test.

It will stop if one test target stops. If your lib tests will detect the hang (without hanging themselves) then the integration tests will not run (and so will not hang).

I did indeed find an invariant to check, and now those mutants are caught. This has the added benefit of failing fast, rather than relying on timeouts!

Either way, cargo-mutants is already helping me a lot to improve the way I test and think about testing!

I'm so glad.