Contributing New Tests

To contribute a new test to the testbed, follow the steps below:

1. Create an issue with a descriptive title of the test that includes the name of the ruleset you are testing.
2. Create a new development branch based on the issue.
3. If the ruleset already exists in the default-validator-stages.xml file (it should), then:
   1. Create a development branch that comprises of two distinct commits:
      1. Adding the tests for the new ruleset. This lets a reviewer run those and observe the fleet members who succeed on these tests vs. those who fail. A template Git commit message for this could be: git commit -m "Add tests for checking <something>" e.g. Add tests for checking schema validation
      2. The ruleset change which fixes the tests so that all fleet members succeed. A template Git commit message for this could be: git commit -m "Fix <what> in <rule-set-name> ruleset <for?> <description?>" e.g. Fix multi-predicate support in check_mdiop ruleset
4. If the ruleset does not already exist in the default-validator-stages.xml file, then: 2. Create a development branch that comprises of three distinct commits:
   1. Adding the calling of the ruleset to the default pipeline in default-validator-stages.xml, without adding any explicit tests. If this is a separate commit, it lets a reviewer check that adding the new ruleset does not break any of the checks for other rulesets.
   2. Adding the tests for the new ruleset. This lets a reviewer run those and observe the fleet members which succeed on these tests vs. those that fail.
   3. Finally, the change to the ruleset which fixes it so that all fleet members succeed.
5. Once complete, create a PR from the branch for review.