Skip to content

Trials

Jump to bottom
Martin Pool edited this page Aug 6, 2022 · 13 revisions

fs_at

2022-08-05, https://github.com/rbtcollins/fs_at 21a7b61c9dbdfdf890fc7d556e04d9150fc3aaf9 using cargo-mutants 0.2.10-pre.

It worked first try, and it finds some gaps, but they're all in Windows implementations, when I'm testing on Linux.

https://gist.github.com/7ec27bceea3fd27cf729310e195ff9b8

This was basically a success showing the crate's well-tested, at least on Linux.

Detecting to skip the Windows modules automatically would have been ideal, although that's complicated a bit by the fact that it uses cfg_if not simple cfg attrs. (See https://github.com/rbtcollins/fs_at/pull/8.)

Just adding #[mutants::skip] into the Windows modules would make it succeed on Linux but then mutation testing would not be useful on Windows.

Conclusions:

time

2022-08-05, https://github.com/time-rs/time using cargo-mutants 0.2.10-pre.

This failed tests in the clean tree, and when I investigated I saw that it also fails in a clean checkout. It turns out (https://github.com/time-rs/time/issues/497) that the tests fail unless you use cargo test --all-features.

I tried cargo mutants -- --all-features, but it still failed, because the extra arguments to cargo mutants are passed only to cargo test, and this needs to go to cargo build --tests as well, or the build fails.

It seems a bit unfriendly to new contributors that cargo test won't pass by default, but it's not unreasonable.

In any case there's a general issue about supporting crates with features: people might want to test mutations with non-default features on.

Conclusions:

im-rs and imbl

2022-08-05, https://github.com/bodil/im-rs, using cargo-mutants 0.2.10-pre.

Seems like it should be hermetic, and it has no workspace, so 🤞.

I see this has been apparently abandoned and forked to https://github.com/jneem/imbl so perhaps that's really the thing to test.

It finds 830 mutants and on my laptop is predicting it'll take three hours to try them all. But it does seem to be proceeding and finding some gaps. Some look like they might be real test coverage gaps.

Partial output: https://gist.github.com/sourcefrog/5475f9695e4a42e97c19551d1d79f42b

; nice cargo mutants -d ~/src/imbl
Freshen source tree ... ok in 14.316s
Copy source and build products to scratch directory ... 559 MB in 0.489s
Unmutated baseline ... ok in 15.745s
Auto-set test timeout to 78.5s
Found 836 mutants to test
src/vector/mod.rs:1904: replace <impl Sum for Vector<A>>::sum -> Self with Default::default() ... NOT CAUGHT in 22.992s
src/hash/set.rs:939: replace <impl From for HashSet<A, S>>::from -> Self with Default::default() ... NOT CAUGHT in 17.512s
src/ord/set.rs:113: replace <impl BTreeValue for Value<A>>::ptr_eq -> bool with true ... NOT CAUGHT in 17.951s
src/ord/map.rs:1220: replace OrdMap<K, V>::symmetric_difference_with -> Self with Default::default() ... NOT CAUGHT in 18.657s
src/hash/map.rs:1834: replace <impl Iterator for IterMut<'a, K, V>>::next -> Option<Self::Item> with Default::default() ... NOT CAUGHT in 18.366s
src/ord/set.rs:272: replace OrdSet<A>::ptr_eq -> bool with false ... NOT CAUGHT in 21.529s
src/ord/set.rs:487: replace OrdSet<A>::insert -> Option<A> with Default::default() ... TIMEOUT in 84.335s
src/tests/mod.rs:5: replace code_fmt -> String with "".into() ... NOT CAUGHT in 20.769s
src/hash/set.rs:796: replace <impl Debug for HashSet<A, S>>::fmt -> Result<(), Error> with Ok(Default::default()) ... NOT CAUGHT in 17.620s
interrupted
Error: interrupted

Conclusions:

  • Let it finish running (on a bigger machine) and investigate whether any are true test coverage gaps.
  • The experiments often end up using only a single core, so this is another case where running multiple mutants in parallel would be useful https://github.com/sourcefrog/cargo-mutants/issues/39.
Clone this wiki locally