Accuracy Test
This page is up-to-date as of Parallax version 0.28.2
An Accuracy Test in Parallax is a set of correspondence points between object space (3D) and stereo image space (4D) which allows us to assess the accuracy of our calibrations and transforms by using the motor coordinates of our translation stage as the ground-truth object points. Accuracy tests can be generated, analyzed, saved and loaded using the Accuracy Testing Tool. Accuracy Tests are stored on disk as Numpy (.npy) files.
To launch the Accuracy Testing Tool, from the Main Window, navigate to:
Main Window Tools > Accuracy Testing Tool
To generate a new accuracy test, navigate to the "Run" tab of the Accuracy Testing Tool:
This window will initiate a routine in which a stage/probe is moved through a series of random positions within some extent, stopping at each point for user input. Here you can choose the settings for this routine:
- Select Stage: Select the stage you want use in the Accuracy Test, from the drop-down via serial number
- Number of Points: The number of positions the probe will stop at. The higher the number, the more reliable your statistics will be. The default value of 100 points typically takes about 15 minutes to complete.
- Extent: the side length of the cube which bounds the stage motion. It should be large enough to cover most of your cameras' fields of view, without extending past them.
- Origin: the center position of the cube which bounds the stage motion. Click "Set current position as origin" to set the selected stage's current position as the center
When you are ready (make sure there are no obstacles for the probe to collide with!) click "Run". The application will move the stage to randomly (uniformly distributed) points within the extent, and stop at each point with a prompt in the Message Log:
In the View Ports, left-click on the probe tip. Try to be as accurate as possible; zooming in is recommended:
(TODO probe tips)
Press "C" on your keyboard to continue the Accuracy Test routine. At each point, select the probe tip in both View Ports and press "C". Once all the test points have been completed, Parallax will automatically save the data to a time-stamped filename in the data directory (accutest_YYMMDD-HHMMSS.npy).
To analyze the data from an Accuracy Test, go to the "Analyze" tab of the Accuracy Testing Tool. Click "Load" to open a file dialog, and select the Accuracy Test file you wish to analyze. Then, select the calibration you wish to assess - the drop-down menu will be auto-populated with the currently loaded calibrations from the Calibration Panel. The selected calibration will be applied to the image points in the data file (i.e. they will be triangulated) and the resulting object points compared the stage positions which serve as "ground truth".
Click "Update" to update the plots:
The widget shows 6 plots in a 2x3 grid. The top row shows histograms of the error values (predictions minus ground truth) for the X, Y, and Z dimensions individually (dx, dy, dz). The bottom row contains interactive 3D scatter plots with visualize the distribution of test points in 3D space. The points are colored according to their error values (again, dx, dy, dz from left to right). This can help assess whether there is a trend towards higher errors (perhaps in one dimension only) as we attempt to triangulate over various regions of the field of view.
The RMS error and the maximum Euclidean error (ds = sqrt(dx^2 + dy^2 + dz^2)), are shown at the bottom of the Analyze Tab:
Optional: You may also wish to apply a transform to the Accuracy Test data. To do this, select a transform from the drop-down menu (it will be auto-populated with the currently loaded transforms from the Transform Panel). Then click Update; the transform will be applied to the object points output from the calibration, and then the transformed points will be compared with the ground truth. This may be necessary if your camera calibration is in a different coordinate system than your probe coordinates.
Hint: By loading multiple calibrations and transforms into the Accuracy Testing tool and analyzing your test points, you can compare which combination of elements would perform best in your reconstruction pipeline.