NEP15 TESTCASE statement

This proposal introduces a new statement TESTCASE which is designed for implementing test cases.

Motivation

Historically the ASSERT statement has been used for implementing test cases. However, it has a few drawbacks when used for this purpose:

• Assertions can be turned off at runtime
• The failure message printed when an assertion fails is too detailed for test cases
• There is no convenient way to test that an exception is raised
• The ASSERT statement is likely to change in the future to accept only expressions without side effects

Proposal

The TESTCASE statement works similarly to ASSERT except it can take an optional EXPECT clause to indicate an expected exception. Its grammar is:

TESTCASE expression [EXCEPT exception]

where the [] indicates an optional clause. When the EXCEPT clause is not present, the expression must be convertible to type Boolean. However, when the EXCEPT clause is present, then the expression may be of any type (including no type at all, such as a call to a function that does not return a value).

The TESTCASE statement always executes and cannot be "turned off".

When a TESTCASE fails, an informative message is printed to standard error, and the program immediately exits with exit code 1. This allows direct integration with any existing test framework.

Examples

A simple case:

TESTCASE math.sqrt(4) = 2

Testing for a required exception:

TESTCASE math.sqrt(-1) EXPECT ValueRangeException