OAS Validation: false-positives using allOf with $refs

[Drew-Kimberly]

Context

We use Prism for API contract testing. I've found that some of our APIs have diverged from their OAS due to false-positives in our tests. In this case, API components defined using a combination of allOf and $ref result in an incongruent JSON Schema document used to validate request/response objects. This leads to false-positive validations and waning confidence in our tests.

Digging in a bit, request/response body schemas that contain allOfs are collapsed here using @stoplight/json-schema-merge-allof. From that repo's docs:

If one of your schemas contain a $ref property you should resolve them using a ref resolver like json-schema-ref-parser to dereference your schema for you first. Resolving $refs is not the task of this library.

And there we have our issue! The allOf merge is performed implicitly inprism-http using a transitive dependency. It needs to be the responsibility of prism-http to dereference the operation's schema prior to collapsing the allOf in order to avoid false-positive validation results.

Current Behavior

$refs are not appropriately handled when contained within allOf collections. Validation passes for request/response bodies despite containing invalid shape according to the spec.

Expected Behavior

Validation fails given invalid req/res data according to the spec, regardless of whether the spec component is defined using allOf and $refs.

Possible Workaround/Solution

We have a workaround where we dereference the spec via @stoplight/json-schema-ref-parser prior to parsing a collection of IHttpOperations.

Steps to Reproduce

The following test case in body.spec.ts illustrates the issue.

it('handles $refs', () => {
      const schema: IMediaTypeContent = {
        id: faker.random.word(),
        mediaType: 'application/json',
        schema: {
          allOf: [
            {
              $ref: '#/definitions/Ref1',
            },
            {
              $ref: '#/definitions/Ref2',
            },
          ],
          definitions: {
            Ref1: {
              type: 'object',
              required: ['ref1'],
              properties: {
                ref1: { type: 'string' },
              },
            },
            Ref2: {
              type: 'object',
              required: ['ref2'],
              properties: {
                ref2: { type: 'number' },
              },
            },
          },
        },
      };

      const actual = validate(
        {
          ref1: 'foo',
          ref2: 'not-a-number',
        },
        [schema],
        ValidationContext.Output,
        'application/json'
      );

      // False-positive validation: Left expected, got a Right
      assertLeft(actual, error =>
        expect(error).toContainEqual(
          expect.objectContaining({ code: 'type', message: 'Response body property ref2 must be number' })
        )
      );
    });

The normalized JSON Schema used for validation is:

    {
      definitions: {
        Ref1: { type: 'object', required: [Array], properties: [Object] },
        Ref2: { type: 'object', required: [Array], properties: [Object] }
      },
      '$ref': '#/definitions/Ref1'
    }

This normalized schema is valid here but incongruent with the input schema where allOf is used with $refs.