Visual Regression tests doc issues

[mscuthbert]

Issue on: https://github.com/0xfe/vexflow/wiki/Visual-Regression-Tests

There does not seem to be an 'npm run generate' command -- there are however various gruntfile commands. Not sure the exact best order to run.

[gristow]

@ronyeh -- putting this here for anyone who searches this as I did. I'm pretty sure the way I got it to work is not the happiest path, or I'd update the docs myself, but it will work:

# commit or stash your changes, or do the first half of this before you start work.
git checkout master 
grunt reference
grunt generate:reference

# now go over to your branch, or pop your changes
git checkout {your_branch_name}
grunt webpack:prodAndDebug
grunt build:esm 
grunt generate:current
grunt diff:reference
open ./build/images/diff

Note that if you run grunt by itself or grunt watch on your branch after generating the reference images from the upstream master, it will clean the build directory entirely, deleting the reference images, and you'll have to start the process above all over. (I've taken to commenting out the 'clean:build' line in the grunt watch task to make that easier.)

[ronyeh]

Thanks for reporting these issues. We definitely need to make this version easier to test. Will look into it!

Just to clarify:

This is for when you want to contribute something new or tweak the visuals, but want to quickly see how your changes affect the current main revision.

Ideally you can do something like:

npm run visual-diff

And it'll do all the required steps to compare your current repo against the master/main revision.

But for now, it would be nice to figure out the exact minimal sequence of npm run / grunt commands to run to get a visual comparison.

[ronyeh]

clean the build directory entirely, deleting the reference images, and you'll have to start the process above all over. (I've taken to commenting out the 'clean:build' line in the grunt watch task to make that easier.)

Hi @gristow and @mscuthbert

Thanks for pointing out these issues. We definitely need to update the wiki docs with the current process for doing visual diffs.

In the Gruntfile.js there is this option:

grunt test:reference:cache

The npm script has the same name.

npm run test:reference:cache

This will back up the reference images before running the build (which deletes the reference images 👎 ). After the build completes, it restores the reference images before running the visual diff.

I am sure there is a better way to do this, which might involve being more careful about NOT deleting reference images.

[ronyeh]

Added some more NPM scripts:
#1581

[ronyeh]

Here's one path to do visual regression testing against a release that is hosted on NPM.

# Work on a crazy new feature.
git branch crazy-clef
... write some code to make clefs 3X bigger ...

# Build the current version.
grunt


# Do a visual comparison against the version 4.2.2 release.
# Download the release from npm.
grunt get:releases:4.2.2

grunt generate:current
grunt generate:release:4.2.2
grunt diff:ver:4.2.2

RIght now, it generates the before & after PNGs. There seems to be a bug with producing the actual diff images... so we will have to investigate that.