Skip to content

Commit

Permalink
docs: add verifications guide
Browse files Browse the repository at this point in the history
  • Loading branch information
p0deje authored and sh3pik committed Nov 11, 2024
1 parent 1843b12 commit 666dfc5
Show file tree
Hide file tree
Showing 5 changed files with 77 additions and 0 deletions.
4 changes: 4 additions & 0 deletions astro.config.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,10 @@ export default defineConfig({
label: "Do Actions",
slug: "docs/guides/actions",
},
{
label: "Check Verifications",
slug: "docs/guides/verifications",
},
],
}
]
Expand Down
Binary file added src/assets/check-tasks-concrete.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added src/assets/check-tasks-generic.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added src/assets/todo-one-checked.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
73 changes: 73 additions & 0 deletions src/content/docs/docs/guides/verifications.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
---
title: Check Verifications
description: How to tell Alumnium to verify
---

Alumnium can verify statements on the web page when you instruct it to **check** something. It analyzes the current state of the web page, optionally including its screenshot, and decides whether the verification successfully passes or fails.

For example, after performing a Google search for "Selenium", you might want to check that search results contain a link to the Selenium browser automation tool:

```python
al.check("selenium.dev is present in the search results")
```

If the search results don't contain a link to `selenium.dev`, Alumnium raises an assertion error and explains:

```bash wrap
AssertionError: The search results do not include 'selenium.dev'. The results shown in the ARIA tree include links to 'foobar2000', 'Wikipedia', and other related topics, but there is no mention of 'selenium.dev'.
```

### Specific Verifications

Similarly to [actions][1], Alumnium works better when the verifications are concrete.

For example, if you are writing a test for completing tasks in the To-Do application and you don't check the exact task's state, you might end up with contradictory verifications passing at the same time!

```python
al.check("task is completed")
al.check("task is not completed")
```

![A screen recording of Alumnium false positive when 1 out of 2 tasks is completed](../../../../assets/check-tasks-generic.gif)

To avoid false positives in the tests, write more concrete verifications:

```python
al.check("task 'buy bread' is completed")
al.check("task 'buy milk' is not completed")
```

![A screen recording of Alumnium passing when 1 out of 2 tasks is completed](../../../../assets/check-tasks-concrete.gif)


[1]: /docs/guides/actions#specific-instructions

### Vision

:::note
Vision verifications are significantly more expensive and slower, so resort to them only when needed.
:::

Occasionally, the web page state is not enough for Alumnium to perform the check. In this case, instruct it to take a screenshot of the page and include it in the verification decision. This is useful when you need to check the visual representation of elements or their spatial relationships.

For example, in your test for the To-Do application, you might need a check that a completed task is shown with a strikethrough style.

![A screenshot of To Do application with one task completed](../../../../assets/todo-one-checked.png)

```python
al.check("'buy bread' title font style is strikethrough")
```

Without a screenshot, this assertion can fail because there is no indication of font style in the web page itself:

```bash wrap
AssertionError: The ARIA tree does not indicate that the 'buy bread' item has a strikethrough style. It shows that the checkbox for 'buy bread' is checked, which typically indicates that the item is completed, but there is no mention of a strikethrough style in the provided ARIA tree.
```
To make the check more reliable, add a screenshot to it:
```python
al.check("'buy bread' title font style is strikethrough", vision=True)
```
Keep in mind, that the screenshot is taken for the visible part of the page.

0 comments on commit 666dfc5

Please sign in to comment.