Rework documentation for clarity, add overrides

README.md

@@ -31,7 +31,50 @@ docker compose exec driver rake validate_all
 
 Output from the command will appear in your terminal.
 
-## Development Mode
+## Terminology
+
+The production UK federation tooling performs *checks* on metadata.
+
+The purpose of this testbed is to perform *tests* on those checks.
+
+Each test is composed of an XML *test file* and an optional
+sidecar *options file* controlling the execution of the test.
+
+A test is executed by sending the *test data* from the test file to
+one or more *endpoints* providing a metadata validation web service.
+
+Each addressed web service runs a requested *validator* on the test data.
+This is a named sequence of MDA stages; the result is a possibly empty
+set of *status metadata* which is returned to the testbed driver.
+
+A test *succeeds* if the returned status metadata matches the
+test's expected results. The test *fails* if any differences
+are detected.
+
+## Endpoints
+
+The `docker compose` deployment currently includes four containers
+implementing a fleet of metadata validation web services which are referenced
+inside the testbed as the following named *endpoints*:
+
+- `v09x` represents the current production deployment. Its name reflects
+  that environment's composition: v0.9 of the Shibboleth Metadata
+  Aggregator, and the Xalan XSL processor included in the classpath.
+- `v09` is identical to `v09x` with the exclusion of the Xalan processor.
+  Because this can run neither the old nor new versions of the URL checking
+  code (the old system involves a Xalan extension, while the new system only
+  exists under MDA v0.10), we do not expect to ever use this configuration
+  in production.
+- `v010x` is anticipated to represent the next step in the evolution of
+  the UK federation tooling, switching out MDA v0.9 for MDA v0.10 but
+  retaining Xalan for compatibility until we are certain that all of the
+  XSL in the production deployment behaves the same under the XSL processor
+  included in the JDK.
+- `v010` is, for now, the anticipated final state for the UK federation
+  deployment once all compatibility issues are resolved so that Xalan can
+  be removed.
+
+## Development mode
 
 The validator fleet has one distinguished member representing the current
 production deployment, currently `v09x`. This validator's port 8080 is
@@ -51,38 +94,61 @@ to determine which validators are available:
 This arrangement allows the test framework itself to be developed
 without having to rebuild the `driver` image for each change.
 
-## Test Setup
+## Configuring tests
+
+Individual unit tests are created by adding files to suitable subdirectories
+of the `tests` directory, with specific subdirectories representing different
+types of test, with the organisation of directories below that being determined
+by the test type.
+
+For example, XML tests (currently the only type supported by the testbed) are
+placed in inside `tests/xml/` in subdirectories representing the rules being
+tested.
 
-Individual unit tests are added to suitable subdirectories of `/tests/`. Each 'type' of unit test *should* go into its own subdirectory (or subdirectory hierarchy), for example, XML tests inside `/tests/xml/`. Tests can be executed against the fleet of validators by running the command `rake validate_all` inside the `driver` service.
+Tests can be executed against the fleet of validators by running the command
+`rake validate_all` inside the `driver` service.
 
-### XML Test Setup
+### XML test configuration
+
+The testbed treats every file with an `.xml` extension under the `tests/xml/`
+directory as a separate XML test. These test files will normally contain
+a single `EntityDescriptor` element representing a SAML entity.
 
-Unit tests that take XML elements (e.g. an `EntityDescriptor`) need to be
-placed in the `/tests/xml` directory and have a `.xml` file extension.
 Running `rake validate_all` from the `driver` service will process all
 test files using the `xml_tests.rb` module against the `default` validator fleet.
 
-The validators to run can be configured via a sidecar YAML file (see Individual Test Configuration).
+## Test options
 
-## Individual Test Configuration
+Each test is configured through named *options*. Each option has a defined
+default which is used when the option is not otherwise specified.
 
-The execution options and the expected outcome of individual tests can be configured in a sidecar YAML file.
-The YAML file needs the same name as, and must be co-located with, the test. For example:
+The available options, and their defaults, are:
 
-```bash
-/tests/<type>/<test-name>.<ext>
-/tests/<type>/<test-name>.yaml
-```
-
-The YAML file can configure the following test options:
+- The `expected` result of the test. This is an array of objects representing
+  MDA `StatusMetadata` item metadata. Each includes:
 
-- The `expected` result of the test. This includes:
-  - The result `status`, which is one-of, `error`, `info`, or `warning`.
+  - The result `status`, which is one of: `error`, `info`, or `warning`. These
+    correspond to `ErrorStatus`, `InfoStatus` and `WarningStatus` respectively.
   - The `component_id` of the component that generated the result.
   - The result description `message`.
+
+  Note that `expected` is *ordered*; a test will be regarded as having failed if
+  the expected results are returned in an unexpected order.
+
+  The default value for `expected` is an empty array, implying that by default
+  validation is expected to succeed.
+
 - Which `validators` to run. If none are specified, the `default` validator is used.
+- Whether to `skip` the test. The default for this option is `false`.
+
+To specify options for a test, create a sidecar YAML file alongside the test file.
+For example, the options file for the test file `tests/xml/example.xml` would
+be `tests/xml/example.yaml`. Note that the alternative `.yml` extension is not
+currently supported.
+
+If the options file does not exist, all options will take their default values.
 
-An example sidecar YAML options file is show below:
+An example options file is shown below:
 
 ``` yaml
 expected:
@@ -94,11 +160,55 @@ validators:
   - validator_2
 ```
 
-If a validator produces an "error", "info", or "warning" result for a given test
+If a validator produces an `error`, `info`, or `warning` result for a given test
 that is not described by that test's `expected` configuration option, the test
 will fail. The actual result and expected result are equal if the String value
 of their `status`, `component_id`, and `message` match exactly.
 
+## Option overrides
+
+Most tests have the same expected results when performed in all of the environments
+represented by the configured endpoints. For situations where this is not the case,
+the testbed supports *option overrides*.
+
+To override an option in some specific situation, include an `override` option
+in the tests's options file. This should be an array of objects each describing
+a situation in which an override is required to the options defined at the top
+level of the file.
+
+Each override object contains keys describing the situation in which the override
+should occur, and keys representing the options to be overridden. The testbed
+processes these overrides in order when looking for the value for an option:
+the first matching override which overrides the option will be taken; if no
+overrides match or the option is not found, the option value will be taken from
+the top level or from the defaults if that is not specified.
+
+The testbed currently supports matching overrides on the basis of endpoint name
+only, using the `endpoint` key. This may either be a string representing a single
+endpoint name, or an array of such names.
+
+The following example shows use of the override feature:
+
+```yaml
+expected:
+  - status: error
+    component_id: component_identifier
+    message: "expected error message"
+override:
+  - endpoint: v09
+    skip: true
+  - endpoint: [v09x, v10x]
+    expected: []
+```
+
+This example means:
+
+- By default, expect an endpoint to return a single error status for this
+  test.
+- On the `v09` endpoint, skip the test entirely.
+- On the `v09x` and `v10x` endpoints, expect no status metadata to be
+  generated (expect the test data to be successfully validated).
+
 ## Copyright and License
 
 The contents of this repository are Copyright (C) the named contributors or their