Trials

fs_at

2022-08-05, https://github.com/rbtcollins/fs_at 21a7b61c9dbdfdf890fc7d556e04d9150fc3aaf9 using cargo-mutants 0.2.10-pre.

It worked first try, and it finds some gaps, but they're all in Windows implementations, when I'm testing on Linux.

https://gist.github.com/7ec27bceea3fd27cf729310e195ff9b8

This was basically a success showing the crate's well-tested, at least on Linux.

Detecting to skip the Windows modules automatically would have been ideal, although that's complicated a bit by the fact that it uses cfg_if not simple cfg attrs. (See https://github.com/rbtcollins/fs_at/pull/8.)

Just adding #[mutants::skip] into the Windows modules would make it succeed on Linux but then mutation testing would not be useful on Windows.

Conclusions:

• Add an --exclude file option which could be used to skip whichever modules don't apply on the platform running the test. https://github.com/sourcefrog/cargo-mutants/issues/58
• Auto-interpretation of cfg would be good though might also be complicated. #50

time

2022-08-05, https://github.com/time-rs/time using cargo-mutants 0.2.10-pre.

This failed tests in the clean tree, and when I investigated I saw that it also fails in a clean checkout. It turns out (https://github.com/time-rs/time/issues/497) that the tests fail unless you use cargo test --all-features.

I tried cargo mutants -- --all-features, but it still failed, because the extra arguments to cargo mutants are passed only to cargo test, and this needs to go to cargo build --tests as well, or the build fails.

It seems a bit unfriendly to new contributors that cargo test won't pass by default, but it's not unreasonable.

In any case there's a general issue about supporting crates with features: people might want to test mutations with non-default features on.

Conclusions:

• An option to pass arguments to cargo build as well as cargo test (#59).
• Perhaps specific handlin of --all-features etc (#60).