[Feature] Run test conditionally

[karentu2]

Problem Statement

For example, I want to run a chainsaw test if a Kyverno policy is in Enforce mode, but if the Kyverno policy is in Audit mode, then the chainsaw test is not run.

Solution Description

One way I can think of to implement this is to add some precondition check field under spec, in the example below I named it precheck. So if the precheck fails, then the test is skipped.

apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
  name: example
spec:
  precheck:
  - try:
    - assert:
        file: path/to/kyverno-policy-enforce-mode.yaml # where this file indicates that a kyverno policy is in Enforce mode
  steps:
  - try:
    - apply:
        file: path/to/apply.yaml
    - assert:
        file: path/to/assert.yaml
    catch: []
    finally: []

Alternatives

It might be better to have the precheck field available on both the spec level so that the entire test can be skipped, and available on the spec.steps level so that specific steps can be skipped.

Additional Context

No response

Slack discussion

https://kubernetes.slack.com/archives/C067LUFL43U/p1712276658755649

Research

• I have searched other issues in this repository and mine is not recorded.

[mveitas]

I want to run a chainsaw test if a Kyverno policy is in Enforce mode, but if the Kyverno policy is in Audit mode, then the chainsaw test is not run

When we rollout a policy, we do this by our environment gradually (dev, test, and production) and control the value via helm values that are passed in. During our testing pipeline, we have added a script to ensure that Enforce mode is always set making our testing easier.

[IshwarKanse]

We would like to see this feature too. Some of our tests depend on the cluster env, cloud provider and auth types. For example, we are currently running these tests as below.

# Get the platform type
dt_platform_type=$(oc get infrastructures cluster -o=jsonpath='{.status.platformStatus.type}')
echo "Platform is $dt_platform_type"

# Check if the cluster is STS or WIF cluster
dt_wif_or_sts=$(oc get authentication cluster -o=jsonpath='{.spec.serviceAccountIssuer}')
echo "$dt_wif_or_sts"

if [[ "$dt_platform_type" == "AWS" && -n "$dt_wif_or_sts" ]]; then
    chainsaw test \
        --config .chainsaw-openshift.yaml \
        --report-name "junit_tempo_aws-sts" \
        --report-path "$ARTIFACT_DIR" \
        --selector type=aws-sts \
        --report-format "XML" \
        --test-dir tests/e2e-openshift-object-stores
else
    echo "Cluster is not AWS STS cluster, skipping the AWS STS tests"
fi

It would be great if this use case is handled in Chainsaw like if all the conditions match, proceed with the test run else skip the test case.

[camaeel]

Cli has parameters: --exclude-test-regex and --include-test-regex.
You can have a script around chainsaw that takes those from a file.

[rahulait]

+1 for this feature request. We would also like to sometimes skip steps within tests. Currently, there is ability to skip the entire test, but not a step within the test. So something like:

apiVersion: chainsaw.kyverno.io/v1alpha2
kind: Test
spec:
  steps:
    - name: step 1
      ....
    - name: Some step we want to skip conditionally
      skip: <based on some var>
      ...