Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for test retries [sc-20570] #952

Merged
merged 8 commits into from
Jul 28, 2024

Conversation

clample
Copy link
Contributor

@clample clample commented Jul 5, 2024

I hereby confirm that I followed the code guidelines found at engineering guidelines

Affected Components

  • CLI
  • Create CLI
  • Test
  • Docs
  • Examples
  • Other

Notes for the Reviewer

This PR adds support for retrying failed tests with npx checkly test and npx checkly trigger. Users can configure retries by either passing the --retries=<numRetries> flag or by setting retries in their Checkly config file.

Check state changes

Previously the CLI tracked the state of a check run by using the CheckRunId. When MQTT/WebSocket updates were received, the CLI would look up which check it was for based on the CheckRunId.

Since we now have retries, the CheckRunId can be different with each retry of a check. In order to track the check state, this PR switches the CLI to use SequenceId. This is stable across all retries. CheckRunId can then be used to track the progress of a particular run.

In abstract-check-runner.js this means that the PR switches to looking up the sequenceId for incoming MQTT messages and using this for tracking the check state. The abstract-list.ts reporter is also updated to track check state using sequenceId.

This PR also introduces a new check state CheckStatus.RETRIED for indicating that a check is being retried. When a check is retried, we leave it in the CheckStatus.RETRIED state rather than switching it back to CheckStatus.SCHEDULING/CheckStatus.RUNNING. The lifecycle of a check that's retried will then look like: SCHEDULING -> RUNNING -> RETRIED -> FAILED/SUCCESSFUL.

Reporters

GitHub reporter

No changes are made to the GitHub reporter. It will simply show the same pass/fail data that it normally does. We could include the number of retries, but there isn't so much horizontal screen space in the GitHub UI to add another column.

Dot reporter

No changes. Will just indicate passed/failed after all retries are finished.

Json reporter

Includes the results for the last check run. Will also include the number of retries.

CI + List Reporter

Prints out info from any retry attempts and shows retrying checks in the summary. Basically following the Notion doc spec. Will add a video to the PR.

Running

The retry support depends on https://github.com/checkly/checkly-runners/pull/1877. For now, it's only possible to run locally. When running, you also need to manually set CHECKLY_CLI_VERSION=4.8.0 (the planned next version), since the backend relies on this to use the new MQTT topic format.

🟢 It's expected that the tests are failing until the runners PR is released

@clample clample force-pushed the chrislample/sc-20570/support-test-retries branch from 7c90236 to c663617 Compare July 5, 2024 13:24
@clample clample force-pushed the chrislample/sc-20570/support-test-retries branch from c663617 to 4c81af3 Compare July 19, 2024 08:05
@clample clample marked this pull request as ready for review July 19, 2024 08:05
@clample clample force-pushed the chrislample/sc-20570/support-test-retries branch from 4c81af3 to 3f4e659 Compare July 19, 2024 08:08
@clample clample requested a review from ferrandiaz July 19, 2024 09:08
this.emit(Events.CHECK_FINISHED, check)
if (resultType === 'FINAL') {
this.disableTimeout(sequenceId)
this.emit(Events.CHECK_SUCCESSFUL, sequenceId, check, result, testResultId, links)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before, we would pass the testResultId to the reporters at the start of the test session.

With retries, though, we don't actually know the testResultId of the final test result until all of the retries are finished. Need to switch to passing it on the CHECK_SUCCESSFUL event.

The testResultId is used to generate links to the test result in the reporters.

ferrandiaz
ferrandiaz previously approved these changes Jul 24, 2024
@clample clample merged commit 469a09b into main Jul 28, 2024
3 checks passed
@clample clample deleted the chrislample/sc-20570/support-test-retries branch July 28, 2024 13:17
@clample clample mentioned this pull request Jul 29, 2024
6 tasks
@clample clample mentioned this pull request Sep 2, 2024
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants