Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add warnings for duplicate test IDs & misconfigured retry commands #100

Open
wants to merge 11 commits into
base: main
Choose a base branch
from

Conversation

pierrebeaucamp
Copy link
Contributor

  • If Captain encounters a test result that contains a duplicate test, it emits a warning.
  • If Captain retries a test, but the updated test result doesn't contain the test anymore, it emits a warning.

Both warnings can also be configured as hard failures: Either as CLI Flags (--fail-on-duplicate-test-id and --fail-on-misconfigured-retry) or in the config file (as fail-on-duplicate-test-id: true in the suite config and fail-on-misconfiguration: true in the retry config respectively).


Duplicate test identification has an interesting edge-case: Although we can trivially check for duplicates using the test.Matches method, we sometimes match tests using an "identity recipe" instead - these recipes usually only use a subset of available test metadata, which leads to the possibility of test.Matches not identifying a duplicate while an "identity recipe" would.

To work around this, we'll check for duplicates using the identity-recipe as well - however this requires that we know expected identity recipes ahead of time. To make this work, we can now pull these identity recipes from Cloud. Captain will do this if it cannot find a .captain/recipes.json file.

Note: This PR doesn't add any kind of invalidation for the cached recipes. So far, our recipes never had to change. Adding invalidation is not much work should we need to do this in the future.


I ended up refactoring a small part of the retry code: I noticed that we explicitly checked for test.Attempt.Status.ImpliesFailure() in each SubstitudeFor method for each framework. Instead, I've put this check inside the filter function that we pass along.

There are only two places that call SubstitudeFor, so adding test.Attempt.Status.ImpliesFailure() into the filter centralized the logic to these two places (instead of duplicating it across each framework). Furthermore, this allowed me to re-use the filter function to later check for misconfigured retry commands.

The only adverse effect of this change is that I had to update a bunch of test-files as they had a stub filter function previously.

@pierrebeaucamp pierrebeaucamp self-assigned this Feb 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants