Data Publisher docs for understanding how to use ShapeExpressions

[lukehesluke]

Spawned from this comment: openactive/openactive-test-suite#562 (comment)

Documentation of how to use them to generate test data (i.e. how to interpret them)

Dependency

This issue's implementation is dependent on openactive/openactive-test-suite#629. I think it would be worth waiting for that issue to be completed before documenting Shape Expressions, as the current structure is inconsistent until that point.

What are ShapeExpressions and why?

Shape Expressions were introduced in Test Interface issue #1. The purpose is explained in the first section of the description of that issue.

To elaborate a little further on what advantages Shape Expressions give that the opportunity criteria (e.g. TestOpportunityBookable) alone does not: If the exact requirements of each criteria is spelled out in a sufficiently descriptive language (Shape Expressions), then new criteria can be added — or existing criteria modified — and a Booking System will not need to update their Test Interface implementation.

And so, Shape Expressions make it easier to extend and fix Test Suite, without requiring users to extend their own Test Interface implementations.

Example

You can see an example of what the ShapeExpressions part of the POST /test-interface/datasets/:testDatasetIdentifier/opportunities interface should look like by running Test Data Generator with integrationTests.useShapeExpressions set to true. i.e.

NODE_CONFIG="{ \"integrationTests\": { \"useShapeExpressions\": true } }" npm run test-data-generator

An example of this output is here: opportunity-test-data.json

Here is a snippet, which shows some of the ShapeExpression constraints for the TestOpportunityBookable criteria:

{
  "@context": [ /* ... */ ],
  "@type": "ItemList",
  "numberOfItems": 1576,
  "itemListElement": [
    {
      "@type": "ListItem",
      "test:numberOfInstancesInDistribution": 201,
      "item": {
        "@type": "ScheduledSession",
        "superEvent": { /* ... */ },
        "test:testOpportunityCriteria": "https://openactive.io/test-interface#TestOpportunityBookable",
        "test:testOpenBookingFlow": "https://openactive.io/test-interface#OpenBookingSimpleFlow",
        "test:testOpportunityDataShapeExpression": [
          {
            "@type": "test:TripleConstraint",
            "predicate": "https://openactive.io/isOpenBookingAllowed",
            "valueExpr": {
              "@type": "test:BooleanNodeConstraint",
              "value": true
            }
          },
          {
            "@type": "test:TripleConstraint",
            "predicate": "https://schema.org/startDate",
            "valueExpr": {
              "@type": "test:DateRangeNodeConstraint",
              "minDate": "2024-03-19T16:19:56.535+00:00"
            }
          },
          /* ...etc */ 
        ],
        "test:testOfferDataShapeExpression": [
          {
            "@type": "test:TripleConstraint",
            "predicate": "https://openactive.io/openBookingFlowRequirement",
            "valueExpr": {
              "@type": "test:ArrayConstraint",
              "datatype": "oa:OpenBookingFlowRequirement",
              "excludesAll": [
                "https://openactive.io/OpenBookingAttendeeDetails",
                "https://openactive.io/OpenBookingIntakeForm",
                "https://openactive.io/OpenBookingApproval"
              ]
            }
          },
          /* ...etc */
        ],

Goals for this issue

Make the following updates to the Test Interface doc:

• Create a new section after the Test Interface Endpoints section, which details Shape Expressions. Use the information from the Shape Expression detail section of this GH issue description.

  • In this section, also specify that the user needs to set integrationTests.useShapeExpressions to true in their Test Suite config in order to use Shape Expressions.

• Update the POST /test-interface/datasets/:testDatasetIdentifier/opportunities to make it clear that the user can use either of the following methods to generate test opportunity data:

  • Switch on the criteria enumeration value (e.g. TestOpportunityBookable). This is already documented
  • Use Shape Expressions, defined in test:testOpportunityDataShapeExpression and test:testOfferDataShapeExpression. Make it clear that this method is recommended for the reasons mentioned earlier. Link to the new section which details ShapeExpressions.

  Additionally, the Example request in this section should include some example shape expression constraints

• Update [test-interface.jsonld](https://github.com/openactive/test-interface/blob/master/test-interface.jsonld) to add namespace properties, classes and enumaration values as are required to fully descripe the ontology of data used in Shape Expressions (see the Shape Expression detail section to get a full picture of this).

Shape Expression detail

• "Opportunity Specification" is going to being used here to describe the body of the POST /test-interface/datasets/:testDatasetIdentifier/opportunities request
• The full Shape Expression of an Opportunity Specification is defined in two fields:
  • test:testOpportunityDataShapeExpression: Constraints relating to the Opportunity
  • test:testOfferDataShapeExpression: Constraints relating to the Offer.
• Each of those two fields contains an array of Constraints, each of which has @type: test:TripleConstraint. Each of these Constraints defines a data requirement for one field

An example (Opportunity) Constraint:

{
  "@type": "test:TripleConstraint",
  "predicate": "https://schema.org/eventStatus",
  "valueExpr": {
    "@type": "test:OptionNodeConstraint",
    "datatype": "schema:EventStatusType",
    "blocklist": [
      "https://schema.org/EventCancelled",
      "https://schema.org/EventPostponed"
    ]
  }
}

Each Constraint has the following format:

• predicate: The field in the Opportunity or Offer that this constraint pertains to. In the above example, this field is https://schema.org/eventStatus, which can be found in .eventStatus of the Opportunity
  • Another example value that predicate could take is https://openactive.io/isOpenBookingAllowed, which can be found in .organizer.isOpenBookingAllowed for a ScheduledSession, .facilityUse.provider.isOpenBookingAllowed for a FacilityUseSlot or .facilityUse.aggregateFacilityUse.isOpenBookingAllowed for an IndividualFacilityUseSlot
  • There are also some special values that predicate can take which are derived predicates. These will be explained later down. These values are https://openactive.io/validFromBeforeStartDate, https://openactive.io/validThroughBeforeStartDate and https://openactive.io/latestCancellationBeforeStartDate.
• valueExpr - defines the requirement that is being imposed for the field specified by predicate. The data is a "Node Constraint", which can be one of many forms, which will be detailed now:

Different @types of "Node Constraints":

• test:DateRangeNodeConstraint: This field must be a ISO-8601 datetime (i.e. serialized as a string). Properties:
  • minDate (string; OPTIONAL): The minimum value that this datetime may have
  • maxDate (string; OPTIONAL): The maximum value that this datetime may have
  • allowNull (boolean; OPTIONAL): If true (default is false), the value can alternatively be null or undefined
• NumericNodeConstraint: This field must be a number. Properties:
  • mininclusive (number; OPTIONAL): Minimum value (inclusive)
  • maxinclusive (number; OPTIONAL): Maximum value (inclusive)
• test:BooleanNodeConstraint: This field must be a boolean. Properties:
  • value (boolean; OPTIONAL): The field must have this exact value e.g. true
• test:OptionNodeConstraint: This field must be from one of a set of options. This is useful for fields whose values use an enumeration type e.g. https://schema.org/eventStatus. Properties:
  • datatype (string; REQUIRED): The enumeration type that this field must use. e.g. "datatype": "schema:EventStatusType" means that the field must use one of the values from the https://schema.org/EventStatusType enumeration type. At present, this can have one of the following values: schema:EventStatusType, oa:RequiredStatusType, schema:EventAttendanceModeEnumeration, oa:TaxMode, oa:OpenBookingFlowRequirement.
  • allowlist (string[]; OPTIONAL): If included, value must be one of the items in this list
  • blocklist (string[]; OPTIONAL): If included, value must NOT be one of the items in this list
  • allowNull (boolean; OPTIONAL): If true (default is false), the value can alternatively be null or undefined
• test:ArrayConstraint: (similar to test:OptionNodeConstraint, but for array fields) This field must be an array, using items one of a set of options. This is useful for fields whose values use an enumeration type e.g. https://schema.org/eventStatus. Properties:
  • datatype (string; REQUIRED): Same as datatype for test:OptionNodeConstraint
  • includesAll (string[]; OPTIONAL): Value must include all items from this array (similar to allowlist for test:OptionNodeConstraint)
  • excludesAll (string[]; OPTIONAL): Value must exclude all items from this array (similar to blocklist for test:OptionNodeConstraint)
  • minLength (number; OPTIONAL): Minimum length of the array
• test:NullNodeConstraint: This field cannot be present. It must be null or undefined. This type has no properties.

Derived Predicates

As mentioned above, https://openactive.io/validFromBeforeStartDate, https://openactive.io/validThroughBeforeStartDate and https://openactive.io/latestCancellationBeforeStartDate, if used as a Constraint's predicate, are in fact derived predicates. This means that they don't just directly specify a field. Here is an explanation for each:

• https://openactive.io/validFromBeforeStartDate: Refers to the date calculated as startDate - validFromBeforeStartDate. When defined as a DateRangeNodeConstraint, allowNull refers to whether validFromBeforeStartDate. can be null.
• https://openactive.io/validThroughBeforeStartDate: Refers to the date calculated as startDate - validThroughBeforeStartDate. When defined as a DateRangeNodeConstraint, allowNull refers to whether validThroughBeforeStartDate. can be null.
• https://openactive.io/latestCancellationBeforeStartDate: Refers to the date calculated as startDate - latestCancellationBeforeStartDate. When defined as a DateRangeNodeConstraint, allowNull refers to whether latestCancellationBeforeStartDate. can be null.

After updating the Test Interface doc

Please create issues for each of the following, which are outside of the scope of this project, but will make this work more complete:

Developer Docs Issue

• Add docs to the Test Interface part of the Developer Docs, which indicate how to use Shape Expressions (linking to the Test Interface docs which now explain Shape Expressions) to generate test data in controlled mode.
  • Make sure this is linked to from the guidance on Controlled Mode.
• Document, somewhere in the https://developer.openactive.io/open-booking-api/test-suite section, that a Random Mode user can use Test Data Generator (with integrationTests.useShapeExpressions set to true) to generate a specification of what test data to generate in their booking system. And that they can and should use ShapeExpressions to do this (linking to the Test Interface docs which now explain Shape Expressions)
  • Make sure this is linked to from the guidance on Random Mode.

Namespace JSON-LD Issue

Update the https://openactive.io/ JSON-LD specification to include all new classes, enumerations, etc that are required to fully describe the ontology of data used in Shape Expressions.