docs

SYCL CTS Developer Documentation

Table of Contents

• General
  • Directory Structure
  • Test Categories
• Procedures
  • Test Plans
  • Pull Requests
  • Continuous Integration (CI)
    • Disabling Test Categories
    • Updating a SYCL Implementation’s Version
• Coding Guidelines
  • Code Style
  • Copyright Header
• Writing Tests
  • Setting up a Simple Category & Test Case
  • Important Catch2 Concepts
    • Test Case Macros
    • Assertion Macros
    • Providing Context
    • Sections
  • Special Macros & Custom Matchers
    • Compile-Time Disabled Test Cases
  • Testing Optional Features
  • Best Practices
  • New-style vs Legacy Test Cases
  • Reduced and full feature set

This document is intended as a comprehensive reference for developers wanting to contribute to the Khronos SYCL Conformance Test Suite.

✏️	This document is still a work-in-progress. Pull requests are welcome! If you are uncertain how to approach a task while developing for the CTS, and it is not covered by this document, please open an issue.

General

The SYCL CTS is a C++17 application that uses Catch2 as its underlying testing framework.

Directory Structure

After first cloning the repository, you will find the following directory structure:

SYCL-CTS
├── .github
├── ci
├── cmake
├── docker
├── docs
├── oclmath
├── test_plans
├── tests
│   ├── common
│   └── <test categories>
├── tools
├── util
└── vendor

The .github, ci and docker folders contain files related to the continuous integration setup, such as workflow definitions, testing containers and per SYCL implementation test category filters. For more information, see Continuous Integration (CI).

The cmake folder contains helper functions and find modules for all supported SYCL implementations. You may find it helpful to browse these files if you run into problems configuring the CTS for a given SYCL implementation.

oclmath contains reference implementations for many of the math functions provided by the SYCL API. To ensure the correct behavior of the latter, they are compared against these reference implementations, which were borrowed from the Khronos OpenCL CTS.

The test_plans directory contains documents describing in detail how certain SYCL features are to be tested. See Test Plans for more information.

The tests directory lies at the heart of the SYCL CTS. It contains many subfolders, one for each test category. Additionally, it contains a common directory with header-only functionalities that are shared between many or all test cases.

Similarly to tests/common, the util folder contains functionality shared by many of the tests in the CTS. It also contains framework-level functionality that underpins the execution of the CTS, for example device selection logic.

💡

The distinction between util and tests/common is blurry and we may revise this structure in the future. If you are unsure where to put new functionality, consider whether it requires a separate translation unit (.cpp file). If so, move it to util; otherwise check if either of the two folders already contains similar/related functionality, and move it there.

Test Categories

Test cases in the SYCL CTS are grouped into categories. Each folder in the tests directory corresponds to one such category. Each category is comprised of one or more translation units and is compiled into a single test executable, named test_<category>.

💡	Before adding a test case, consider whether there already exists a category it would fit into, or whether a new category is required.

💡	The CTS supports disabling the compilation of entire categories for certain SYCL implementations. See Procedures for more information.

Procedures

✏️	TODO: Explain supporting multiple SYCL implementations (test category filters, compile-time macros).

Test Plans

For SYCL 2020, the SYCL CTS has adopted test plans as a way of planning how a given feature is going to be tested. Using test plans, you can get feedback on your planned testing approach and scope without spending effort on code that may then have to be changed during review.

You can find existing test plans in the test_plans directory.

💡	Consider creating a test plan when tackling a new test category. For smaller contributions, such as the addition of a new test case to an existing test category, the creation of a test plan can usually be skipped.

Pull Requests

The SYCL CTS uses pull requests and code reviews before merging any changes, including test cases, bugfixes and improvements to the testing infrastructure.

Before a pull request can be merged, the following steps need to be addressed:

1. To be able to contribute code to the SYCL CTS, you will have to sign the Khronos Open Source Contributer License Agreement (CLA).

2. All CI checks must be green.

3. A pull request requires at least one review approval from a Khronos member.

   1. If you work for an organization that is a Khronos member, at least one approval from a different member is required.

Merging is done by the author of the pull request or someone from the same organization (for Khronos members), or by @bader or @psalz for third party contributions.

💡	Bumping the version of a SYCL implementation used in CI requires no review approvals if the PR was opened by the respective implementer. See Continuous Integration (CI) for more information.

Continuous Integration (CI)

To ensure that the SYCL CTS remains compatible with all three supported SYCL implementations, a continuous integration (CI) pipeline is run on every pull request. To pass the pipeline, the CTS needs to compile for all SYCL implementations. If this is not feasible, parts of the CTS may have to be compile-time disabled. See Disabling Test Categories and Compile-Time Disabled Test Cases for more information.

❗	The CTS is currently only compiled during CI, but not executed. This means that passing CI does not imply anything about the quality of your testing logic.

Compilation takes place inside of Docker containers, with a separate container used for each SYCL implementation. The container images are available at the Khronos DockerHub repository and the corresponding Dockerfiles can be found in the docker directory.

💡	Using the CTS CI container images locally can be a quick and easy way to spin up a working development environment when debugging an issue for a given SYCL implementation.

Disabling Test Categories

As the CTS and different SYCL implementations are being independently developed, it is not always possible to guarantee that all tests compile for all SYCL implementations. To enable the CI pipeline to discover actual bugs and regressions while ignoring cases that are known to be non-working, the CTS allows to disable the compilation of entire test categories during CMake configuration time.

To disable one or more test categories, simply configure the CTS with the option -DSYCL_CTS_EXCLUDE_TEST_CATEGORIES=<filter-file>, where <filter-file> is a file containing a list of categories to ignore.

A test category filter for each SYCL implementation corresponding to the version currently tested in CI can be found in the ci directory.

💡

While test category filters provide a convenient way of ensuring the CTS passes CI, it can be a heavy-handed approach in scenarios where only some parts of a category don’t compile for a given implementation. To address this issue, the CTS offers finer-grained control over which parts of a test are being compiled through Compile-Time Disabled Test Cases.

Updating a SYCL Implementation’s Version

The version of each SYCL implementation is specified in the GitHub workflow definition file.

The GitHub actions workflow needs to interact with DockerHub to push new Docker images for use in subsequent CI runs. This requires credentials that, for security reasons, are only available to the workflow when it is run on a branch in the main repository, not from a fork.

❗	To update the version of a SYCL implementation, always push the commit to the main CTS repository directly.

💡	After updating the version of a SYCL implementation, the category filters should be regenerated. To do so, simply run ci/generate_exclude_filter.py.

Coding Guidelines

Code Style

The CTS uses clang-format to ensure a consistent coding style. While some parts of the CTS are not yet formatted according to clang-format, all new additions and modifications must be.

❗	Please format your code using clang-format before submitting a pull request. However, make sure to only format the parts that you actually modified (for example using clang-format-diff.py), to avoid noise in your patch.

Copyright Header

Each file in the SYCL CTS should be prefaced by the Khronos copyright header:

/*******************************************************************************
//
//  SYCL 2020 Conformance Test Suite
//
//  Copyright (c) <YEAR> The Khronos Group Inc.
//
//  Licensed under the Apache License, Version 2.0 (the "License");
//  you may not use this file except in compliance with the License.
//  You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
//  Unless required by applicable law or agreed to in writing, software
//  distributed under the License is distributed on an "AS IS" BASIS,
//  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
//  See the License for the specific language governing permissions and
//  limitations under the License.
//
*******************************************************************************/

where <YEAR> refers to the current year when creating a new file, or a range (e.g. 2020 - 2022) when updating an existing file.

Writing Tests

This section contains guidelines on how to write test cases for the SYCL CTS. We recommend that you try and stick to these guidelines, however, they are not to be considered hard and fast rules, and best practices are still being developed.

Setting up a Simple Category & Test Case

To create a new test category create the following files inside the tests directory:

tests
└── simple
   ├── CMakeLists.txt
   └── simple.cpp

In tests/simple/CMakeLists.txt add the following boilerplate:

file(GLOB test_cases_list *.cpp)
add_cts_test(${test_cases_list})

Then in tests/simple/simple.cpp add the following:

#include "../common/common.h"

TEST_CASE("a simple test case", "[simple]") {
    sycl::buffer<int> buf(1);
    sycl::queue queue = sycl_cts::util::get_cts_object::queue();
    queue.submit([&](sycl::handler& cgh) {
        sycl::accessor w{buf, cgh, sycl::write_only};
        cgh.single_task<class simple_kernel>([=] {
            w[0] = 42;
        });
    });

    sycl::host_accessor r{buf, sycl::read_only};
    CHECK(r[0] == 42);
}

This adds a test case with the description "a simple test case" and the tag [simple]. Both can later be used to narrow down the set of test cases that will be executed during runtime.

When configuring CMake, the new test category will automatically be detected and a target with the name test_simple is added. You can run the test case by executing ./bin/test_simple.

❗	For historic reasons, the CTS currently contains many test cases that are written in a different style. Please see New-style vs Legacy Test Cases for more information.

Important Catch2 Concepts

The SYCL CTS relies on Catch2 as its underlying testing framework. This section will list the most important concepts required to write tests with Catch2. For a comprehensive overview of all features, please refer to the Catch2 documentation. In addition, the CTS provides several custom utilities to extend Catch2’s feature set. See Special Macros & Custom Matchers for more information.

Test Case Macros

Catch2 provides several macros of varying complexity for defining test cases. While different macros take different parameters, they all require a description and optionally a list of tags to be specified.

• TEST_CASE is the most basic macro, useful for test cases that deal with APIs that are not templated in any way.

• TEMPLATE_TEST_CASE can be provided with one or more types that are then available as TestType within the test case. The test case is then instantiated separately for each type.

• TEMPLATE_TEST_CASE_SIG can be used to make one or more template parameters (including non-type template parameters) available under a custom name.

💡	Use TEMPLATE_TEST_CASE_SIG("my test", "[my-tag]", ((int D), D), 1, 2, 3) to test APIs that support multiple dimensions. The test case will be executed three times, with D having a value of 1, 2 and 3, respectively.

Assertion Macros

• CHECK(condition) asserts that the provided condition is true. If it is false, the assertion failure will be reported and the test case continues execution.

• REQUIRE(condition) works like CHECK, but will abort the current test case upon failure.

💡	Use CHECK by default, only resort to REQUIRE when further execution of a test case would result in a crash (for example REQUIRE(arr.size() >= 2); if(arr[1] == 123) { /* …​ */ }).

Providing Context

While Catch2 already provides great error reporting out of the box, it can sometimes be helpful to provide additional context alongside a failing assertion.

• CAPTURE(…​) can be used to print the name and value of arbitrary values alongside a failing assertion.

• INFO(message) allows to provide additional information in the form of natural language descriptions. iostream-style formatting is supported.

Example usages of both:

TEST_CASE("my test case") {
    const int x = 3;
    const int y = 4;
    const int z = x * y;
    // Shorthand
    CAPTURE(x, y);
    // More verbose
    INFO("checking that x (" << x << ") times y (" << y << ") equals 20");
    CHECK(z == 20);
}

Sections

Sections provide a way of sharing code between related yet distinct testing logic. For example this test case will be executed twice, once for each section. While only one section is entered each time, setup_something() and tear_something_down() will be executed in both cases:

TEST_CASE("my test case with sections") {
    setup_something();

    SECTION("testing one thing") {
        /* ... */
    }

    SECTION("testing another thing") {
        /* ... */
    }

    tear_something_down();
}

Special Macros & Custom Matchers

The SYCL CTS extends Catch2’s functionality with several custom macros and matchers.

Compile-Time Disabled Test Cases

While writing test cases for the CTS, you may want to test features that have not yet been implemented by all of the SYCL implementations. Test category filters (see Procedures) offer a way of disabling entire test categories for a set of implementations. However this is often too coarse grained of an approach: In many cases, a certain feature may exist partially in an implementation, but may not yet offer all of the APIs prescribed by the specification. Unfortunately, using such missing APIs in test cases (for example constructor overloads or member functions) will then prevent the entire test category from compiling (for the SYCL implementation in question). To allow testing of features that are present, while not compiling those that are missing, the SYCL CTS offers special macros for disabling individual test cases at compile time.

Their usage is best explained in an example:

DISABLED_FOR_TEST_CASE(hipSYCL)("some feature works as expected", "[some-feature]")({
    CHECK(sycl::something_that_hipsycl_does_not_yet_support() == 123);
});

While for other SYCL implementations the test case will compile as if it were a normal TEST_CASE, for hipSYCL it will instead compile to a test case that fails at runtime with the message "This test case has been compile-time disabled.".

Note that unlike the normal TEST_CASE macro, DISABLED_FOR_TEST_CASE requires that the body of the test is wrapped in parentheses and followed by a semicolon.

The CTS currently provides the following macros for compile-time disabling test cases:

• DISABLED_FOR_TEST_CASE(<impls…​>)(<description>, <tags>)(<body>)

• DISABLED_FOR_TEMPLATE_TEST_CASE_SIG(<impls…​>)(<description>, <tags>, <signature>, <types…​>)(<body>)

where <impls…​> is a comma-separated list of hipSYCL and/or DPCPP.

✏️	TODO: Custom matchers.

Testing Optional Features

The CTS may include tests that cannot be executed in all circumstances. Examples of such tests include tests for optional features, tests that depend on certain device capabilities, tests that require multiple devices as well as tests for vendor extensions. In such scenarios, Catch2’s SKIP macro should be used to explicitly report a test case as skipped.

Best Practices

Here is a list of best practices for writing test cases. These are not set in stone and are likely to evolve over time.

• Always write tests using Catch2 macros, avoid legacy test cases.

• Avoid old-style if(!condition) FAIL("reason"); pattern. Use CHECK(condition) instead.

• Keep test cases small and focused to a single concept / behavior. Even a single function could be tested with several test cases.

• Use natural language descriptions for test cases:

  • Avoid: "host_accessor range mismatch exception".

  • Prefer: "host_accessors throws if accessed range exceeds buffer dimensions".

• Tag test cases according to the feature being tested:

  • Use [some_type] for types that exist in the SYCL specification (example: [host_accessor]).

  • Use [some-concept] for concepts without a clearly associated type (example: [backend-interop]).

• Group test cases into files at your own discretion. It is certainly possible to have all test cases for a given API within the same file. However, for larger features distributing test cases across multiple files may be preferable.

• Try to order test cases in a file in the same order as their associated API specification (if possible).

New-style vs Legacy Test Cases

When browsing the CTS, you will likely encounter two different kinds of test cases: New-style test cases and legacy test cases. New-style test cases are written using free-standing Catch2 macros such as TEST_CASE and will look something like this:

TEST_CASE("SYCL feature XY works as expected", "[feature-xy]") {
    // ...
    CHECK(works_as_expected);
}

Importantly, multiple of these test cases will typically be grouped into a single file.

Legacy test cases on the other hand use a class-based approach, where a test case is implemented by extending the sycl_test::util::test_base class. Testing logic is then implemented in the run member function:

#define TEST_NAME feature_xy

namespace TEST_NAMESPACE {
using namespace sycl_cts;

class TEST_NAME : public util::test_base {
public:
  void get_info(test_base::info &out) const override { /* ... */ }

  void run(util::logger &log) override {
      // ...
      if(!works_as_expected) {
          FAIL("feature XY does not work as expected");
      }
  }
};

util::test_proxy<TEST_NAME> proxy;
}

While legacy test cases are still mapped to Catch2 under the hood, they require a lot of boilerplate code and therefore testing logic for distinct aspects of a feature are often grouped into a single test case, making them harder to comprehend and debug. Although technically not required, usually only one class extending test_base is defined per file.

❗	Always write new-style test cases.

Reduced and full feature set

SYCL 2020 specifies a full and a reduced feature set. The full feature set includes all features in the core SYCL specification without exceptions, the reduced feature set makes certain features optional. To ensure that reduced feature set implementations can test conformance, the CTS option SYCL_CTS_ENABLE_FEATURE_SET_FULL is available, which can be set to OFF (it is ON by default). Tests for full features should be conditionally included with #if SYCL_CTS_ENABLE_FEATURE_SET_FULL to compile-time disable these tests for reduced feature set implementations.