Add "invert_result" to test() option

[twied]

This is very similar to "should_fail", but does not change the test results to "Expected Fail" / "Unexpected Pass".

"Expected Fail" and "Unexpected Pass" are useful when they communicate to developers that additional work needs to be done, e.g. as part of Test Driven Development or to demonstrate problems with some edge cases that need to be addressed in the future.

In other cases, a non-zero status code is actually the whole test, e.g. a program under test that is expected to exit with a non-zero status code when given invalid arguments or a bad configuration file. This is the expected behavior and hence should be reported as "Pass", not "Expected Fail".

"invert_result" and "should_fail" can be combined to produce "Unexpected Pass" for non-zero status codes and "Expected Fail" for zero status codes.

test(
    'fail if config file is missing',
    executable(...),
    args: [ '--config', '/path/does/not/exist' ],
    invert_result: true)

[bonzini]

Can you check if #13559 would work for you too?

[bonzini]

This CI failure is relevant, too:

../test cases/common/282 inverted result with should fail/ret0.c:1:5: error: function declaration isn’t a prototype [-Werror=strict-prototypes]
    1 | int main() { return 0; }
      |     ^~~~

[jpakkane]

ISTR that some people have also requested a similar check, but one that checks that the exit code has a specific value. So something like:

add_test(..., returncode: 5)

Would this make more sense and be more general? Having multiple ways of inverting the result is confusing, ideally there should only be one way.

[twied]

Can you check if #13559 would work for you too?

I am not too familiar with TAP and cannot comment on the specifics, but in general I agree with and share the goal of separating the concepts of "some other exit code than zero means success" and "use testresults expected-fail / unexpected-pass instead of pass / fail". I know the expected return code in my use case, so having a success_returncode would work perfectly fine. I do not quite understand why #13559 deprecates / renames should_fail though.

This CI failure is relevant, too:

../test cases/common/282 inverted result with should fail/ret0.c:1:5: error: function declaration isn’t a prototype [-Werror=strict-prototypes]
    1 | int main() { return 0; }
      |     ^~~~

I have a fix for that one (adding int argc, char** argv), but I do not know what causes the other CI failure.

Having multiple ways of inverting the result is confusing, ideally there should only be one way.

I agree. Unless there is a use case for expressing "anything other than 0 is a success" instead of "this return code is a success", this PR can be closed in favor of #13559.

[bonzini]

Thanks @twied! Closing in favor of #13559 then.