chore: update v0.0.7 docs (#527)

Signed-off-by: Charles-Edouard Brétéché <[email protected]>

website/docs/more/kuttl-migration.md

@@ -38,7 +38,6 @@ On invocation, the command:
 - Discovers folders from the specified paths.
 - Reads the files within these folders, specifically looking for YAML files.
 - Inspects each YAML file to check if it's a KUTTL resource. If it is, the command tries to convert it to a Chainsaw resource.
-- The conversion handles two types of KUTTL resources: TestSuite and TestStep. It also reports an error for unsupported resources like `TestAssert`.
 - If the `--save` flag is set, the converted Chainsaw tests are saved to a new file with the extension `.chainsaw.yaml`.
 - If the `--save` and `--overwrite` flags are set, the converted Chainsaw tests are saved to the same file as the original one.
 
@@ -69,5 +68,4 @@ The file path for saving is determined by the `--overwrite` flag; if it is set,
 
 The migration command has the following limitations:
 
-- Converting `TestAssert` resources is not supported (yet)
-- Some fileds in KUTTL `command` are not supported and will raise an error
+- Some fields in KUTTL `command` are not supported and will raise an error

website/docs/tests/catch-finally/index.md

@@ -1,9 +1,10 @@
 # What is catch / finally
 
 Catch and Finally are additional fields to collect certain information about the outcome of a step should it fail (in the case of `catch`) or at the end of the step (in the case of `finally`).
+
 The ultimate goal of collectors is to gather information about the failure of a step and therefore help understand what caused it to fail.
 
-A test step can have an arbitrary number of collectors.
+A test step can have an arbitrary number of collectors. A collector can be configured with a `timeout`.
 
 !!! note
 
@@ -99,3 +100,20 @@ Catch / Finally are a per step configuration and are registered under the `catch
               echo "goodbye"
     ```
     See [Scripts](scripts.md) for details and supported configurations.
+
+!!! example "Execute a custom script with timeout"
+
+    ```yaml
+        try:
+        # ...
+        catch:
+        - script:
+            content: |
+              echo "an error has occured"
+        finally:
+        - script:
+            content: |
+              echo "goodbye"
+          timeout: 15s
+    ```
+    See [Scripts](scripts.md) for details and supported configurations.

website/docs/tests/configuration.md

@@ -124,6 +124,12 @@ chainsaw test --config my-config.yaml --test-dir /path/to/tests --parallel 10
 
 This command will run tests using the configuration from `my-config.yaml`, taking tests from `/path/to/tests`, and running a maximum of `10` tests simultaneously.
 
+## Reports
+
+Chainsaw can generate JUnit reports in `XML` or `JSON` format.
+
+Provide a report format and report name in the configuration or via CLI flags.
+
 ## Reference
 
 Refer to the reference documentations for details about supported fields in the configuration file and/or supported flags in the `test` command.

website/docs/tests/operations/apply.md

@@ -99,11 +99,12 @@ Below is an example of using an [operation check](./check.md#apply).
     # ...
     - apply:
         file: my-pod.yaml
-        check:
-          # an error is expected, this will:
-          # - succeed if the operation failed
-          # - fail if the operation succeeded
-          (error != null): true
+        expect:
+        - check:
+            # an error is expected, this will:
+            # - succeed if the operation failed
+            # - fail if the operation succeeded
+            ($error != null): true
     # ...
     ```
 
@@ -119,10 +120,16 @@ Below is an example of using an [operation check](./check.md#apply).
             name: chainsaw-quick-start
           data:
             foo: bar
-        check:
-          # an error is expected, this will:
-          # - succeed if the operation failed
-          # - fail if the operation succeeded
-          (error != null): true
+        expect:
+        - match:
+            # this check applies only if the match
+            # statement below evaluates to `true`
+            apiVersion: v1
+            kind: ConfigMap
+          check:
+            # an error is expected, this will:
+            # - succeed if the operation failed
+            # - fail if the operation succeeded
+            ($error != null): true
     # ...
     ```

website/docs/tests/operations/check.md

@@ -7,43 +7,79 @@ Considering an operation success or failure is not always as simple as checking
 
 To support those kind of use cases, some operations support an additional `check` field to evaluate the operation result against an assertion tree.
 
+!!! info
+
+    Assertions in Chainsaw are based on **assertion trees**.
+
+    Assertion trees is a solution to declaratively represent complex conditions like partial array comparisons or complex operations against an incoming data structure.
+
+    Assertion trees are compatible with standard assertions that exist in tools like KUTTL but can do a lot more.
+    Please see the [assertion trees documentation](https://kyverno.github.io/kyverno-json/policies/asserts/) in kyverno-json for details.
+
 !!! tip "Checked model"
     Different operation have a different model passed through the assertion tree.
 
+    The object passed to the assertion tree is the output object of the operation. Additional data like error or standard logs are passed using bindings (`$error`, `$stdout`, `$stderr`)
+
+## `Expect` vs `Check`
+
+While a simple check is enough to determine the result of a single operation, we needed a more advanced construct to cover `apply` and `create` operations. Those operations can operate on files containing multiple manifests and every manifest can have a different result.
+
+To support more granular checks we use the `expect` field that contains an array of [Expectation](../../apis/chainsaw.v1alpha1.md#chainsaw-kyverno-io-v1alpha1-Expectation).
+Every expectation is made of an optional `match` and a `check` statement.
+
+This way it is possible to control the scope of a `check`.
+
+!!! tip "Null match"
+    If the `match` statement is null, the `check` statement applies to all manifests in the operation.
+
+    If no expectation matches a given manifest, the default expectation will be used, checking that no error occured.
+
 ## Apply
 
-`apply` supports the following elements to be checked:
+`apply` supports `expect` and has the following elements to be checked:
 
 | Name | Purpose | Type |
 |---|---|---|
-| `error` | The error message (if any) at the end of the operation | `string` |
-| `resource` | The state of the resource (if any) at the end of the operation | `object` |
+| `$error` | The error message (if any) at the end of the operation | `string` |
+| `@` | The state of the resource (if any) at the end of the operation | `object` |
 
 ## Create
 
-`create` supports the following elements to be checked:
+`create` supports `expect` and has the following elements to be checked:
+
+| Name | Purpose | Type |
+|---|---|---|
+| `$error` | The error message (if any) at the end of the operation | `string` |
+| `@` | The state of the resource (if any) at the end of the operation | `object` |
+
+## Delete
+
+`delete` supports `check` and has the following elements to be checked:
 
 | Name | Purpose | Type |
 |---|---|---|
-| `error` | The error message (if any) at the end of the operation | `string` |
-| `resource` | The state of the resource (if any) at the end of the operation | `object` |
+| `$error` | The error message (if any) at the end of the operation | `string` |
+| `@` | The state of the resource (if any) at the end of the operation | `object` |
 
 ## Command
 
-`command` supports the following elements to be checked:
+`command` supports `check` and has the following elements to be checked:
 
 | Name | Purpose | Type |
 |---|---|---|
-| `error` | The error message (if any) at the end of the operation | `string` |
-| `stdout` | The content of the standard console output (if any) at the end of the operation | `string` |
-| `stderr` | The content of the standard console error output (if any) at the end of the operation | `string` |
+| `$error` | The error message (if any) at the end of the operation | `string` |
+| `$stdout` | The content of the standard console output (if any) at the end of the operation | `string` |
+| `$stderr` | The content of the standard console error output (if any) at the end of the operation | `string` |
+| `@` | Always `null` | |
 
 ## Script
 
-`script` supports the following elements to be checked:
+`script` supports `check` and has the following elements to be checked:
 
 | Name | Purpose | Type |
 |---|---|---|
-| `error` | The error message (if any) at the end of the operation | `string` |
-| `stdout` | The content of the standard console output (if any) at the end of the operation | `string` |
-| `stderr` | The content of the standard console error output (if any) at the end of the operation | `string` |
+| `$error` | The error message (if any) at the end of the operation | `string` |
+| `$stdout` | The content of the standard console output (if any) at the end of the operation | `string` |
+| `$stderr` | The content of the standard console error output (if any) at the end of the operation | `string` |
+| `@` | Always `null` | |

website/docs/tests/operations/command.md

@@ -64,6 +64,6 @@ Below is an example of using an [operation check](./check.md#command).
           # an error is expected, this will:
           # - succeed if the operation failed
           # - fail if the operation succeeded
-          (error != null): true
+          ($error != null): true
     # ...
     ```

website/docs/tests/operations/create.md

@@ -103,11 +103,12 @@ Below is an example of using an [operation check](./check.md#create).
     # ...
     - apply:
         file: my-pod.yaml
-        check:
-          # an error is expected, this will:
-          # - succeed if the operation failed
-          # - fail if the operation succeeded
-          (error != null): true
+        expect:
+        - check:
+            # an error is expected, this will:
+            # - succeed if the operation failed
+            # - fail if the operation succeeded
+            ($error != null): true
     # ...
     ```
 
@@ -123,10 +124,16 @@ Below is an example of using an [operation check](./check.md#create).
             name: chainsaw-quick-start
           data:
             foo: bar
-        check:
-          # an error is expected, this will:
-          # - succeed if the operation failed
-          # - fail if the operation succeeded
-          (error != null): true
+        expect:
+        - match:
+            # this check applies only if the match
+            # statement below evaluates to `true`
+            apiVersion: v1
+            kind: ConfigMap
+          check:
+            # an error is expected, this will:
+            # - succeed if the operation failed
+            # - fail if the operation succeeded
+            ($error != null): true
     # ...
     ```

website/docs/tests/operations/delete.md

@@ -21,10 +21,11 @@ Below is an example of using `delete` in a `Test` resource.
       - try:
         # ...
         - delete:
-            apiVersion: v1
-            kind: Pod
-            namespace: default
-            name: my-test-pod
+            ref:
+              apiVersion: v1
+              kind: Pod
+              namespace: default
+              name: my-test-pod
         # ...
     ```
 
@@ -43,9 +44,50 @@ Below is an example of using `delete` in a `TestStep` resource.
       try:
       # ...
       - delete:
+          ref:
+            apiVersion: v1
+            kind: Pod
+            namespace: default
+            name: my-test-pod
+      # ...
+    ```
+
+## Operation check
+
+Below is an example of using an [operation check](./check.md#delete).
+
+!!! example "With check"
+
+    ```yaml
+    # ...
+    - delete:
+        ref:
           apiVersion: v1
           kind: Pod
           namespace: default
           name: my-test-pod
-      # ...
+        check:
+          # an error is expected, this will:
+          # - succeed if the operation failed
+          # - fail if the operation succeeded
+          ($error != null): true
+    # ...
+    ```
+
+!!! example "With check"
+
+    ```yaml
+    # ...
+    - delete:
+        ref:
+          apiVersion: v1
+          kind: Pod
+          namespace: default
+          name: my-test-pod
+        check:
+          # an error is expected, this will:
+          # - succeed if the operation failed
+          # - fail if the operation succeeded
+          ($error != null): true
+    # ...
     ```

website/docs/tests/operations/script.md

@@ -61,6 +61,6 @@ Below is an example of using an [operation check](./check.md#script).
           # an error is expected, this will:
           # - succeed if the operation failed
           # - fail if the operation succeeded
-          (error != null): true
+          ($error != null): true
     # ...
     ```