Skip to content

Commit

Permalink
feat: Application of golden file pattern for testing breaking test er…
Browse files Browse the repository at this point in the history
…rors.

Running the tc39 tests also in "enhanced" (esbuild) compatibility mode.
Previously, the test wrote the JSON format file to be saved to the standard output,
which had to be manually cleaned from the other test output.
This solution is difficult to apply in the case of one or more compatibility modes,
so it has been changed to the usual golden file pattern.

The reference breaking_test_errors-extended.json and breaking_test_errors-enhanced.json
files can be updated using the go test -update command.
  • Loading branch information
szkiba committed May 13, 2024
1 parent b0486e4 commit c5b1c24
Show file tree
Hide file tree
Showing 4 changed files with 213 additions and 22 deletions.
10 changes: 4 additions & 6 deletions js/tc39/README.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,17 @@
# Introduction to a k6's TC39 testing

The point of this module is to test k6 goja+babel combo against the tc39 test suite.
The point of this module is to test k6 goja+babel and k6 goja+esbuild combo against the tc39 test suite.

Ways to use it:
1. run ./checkout.sh to checkout the last commit sha of [test262](https://github.com/tc39/test262)
that was tested with this module
2. Run `go test &> out.log`

If there are failures there will be a JSON with what failed.
The full list of failing tests, and the error, is in `breaking_test_errors.json`. All errors list there with the corresponding error will *not* be counted as errors - this is what the test expects, those specific errors.
The full list of failing tests, and the error, is in `breaking_test_errors-*.json`. All errors list there with the corresponding error will *not* be counted as errors - this is what the test expects, those specific errors.
Due to changes to goja it is not uncommon for the error to change, or there to be now a new error on previously passing test, or (hopefully) a test that was not passing but now is.
In all of those cases `breaking_test_errors.json` needs to be updated. Currently, the output is the *difference* in errors, so we need to "null" the file. To that we set it to an empty JSON `echo '{}' > breaking_test_errors.json`.
Run the test with output to a file `go test &> breaking_test_errors.json`. And then edit `out.log` so only the JSON is left. I personally search for `FAIL` and that should be the first thing just *after* the JSON, delete till the end of file. This is easiest done with sed(or vim) as in `sed -i '/FAIL/,$d' breaking_test_errors.json`.
In all of those cases `breaking_test_errors-*.json` needs to be updated. Run the test with `-update` flag to update: `go test -update`

NOTE: some text editors/IDEs will try to parse files ending in `json` as JSON, which given the size of `breaking_test_errors.json` might be a problem when it's not actually a JSON (before the edit). So it might be a better idea to name it something different if editing by hand and fix it later.
NOTE: some text editors/IDEs will try to parse files ending in `json` as JSON, which given the size of `breaking_test_errors-*.json` might be a problem when it's not actually a JSON (before the edit). So it might be a better idea to name it something different if editing by hand and fix it later.

This is a modified version of [the code in the goja
repo](https://github.com/dop251/goja/blob/master/tc39_test.go)
Expand Down
Loading

0 comments on commit c5b1c24

Please sign in to comment.