Skip to content

Tutorial: Reticle‐to‐Probe Transform from Triangulated Correspondence Points

Jump to bottom
Chris Chronopoulos edited this page Aug 30, 2023 · 6 revisions

This page is up-to-date as of Parallax version 0.33.0

This tutorial walks you through a procedure which is a major raison dêtre for Parallax: establishing a transform from world coordinates (be they headframe, CCF, etc.) to probe coordinates, for an arbitrarily placed probe, without having to manually register correspondence points through probe contact.

For this tutorial you'll need:

  • A calibration checkerboard
  • A targeting reticle, or some target with precisely defined fiducial markers
  • One probe module (Neuropixels probe mounted on a New Scale stage)
  • Two camera modules (FLIR Blackfly cameras with long working distance optics)

Instructions

  1. Set up your rig

    a. Mount the targeting reticle at the center of your rig. On the Modular Insertion System (MIS), this will be the headframe mount at the center of the table.

    b. Mount your cameras so they are pointing at the reticle. They should point from different angles - 90 degrees is ideal for triangulation accuracy, but smaller angles are also acceptable.

    c. Make sure the probe can be positioned so that it reaches the reticle and can navigate its extent

    d. Arrange a way hold the calibration checkerboard in both cameras' fields of view simultaneously, while allowing for articulation of the checkerboard through various angles. We suggest two optical posts joined with a right-angle connector.

PXL_20230812_044219569

  1. Start up Parallax.

  2. Right-click each of the two Viewports in the Main Window, and select Parallax > Cameras > (camera serial number) for each camera. Which camera streams to which Viewport is not important, but it must be consistent throughout this procedure.

  3. Focus your cameras so that they have as much of the targeting reticle in focus as possible.

  4. Place the checkerboard within view of both cameras. It should be entirely visible, and mostly in focus, in both views. Ideally it should span a fraction (~1/3) of the camera field of view, so that we can move it around to collect corner points.

checkerboard

Don't adjust the camera focus for the checkerboard - you want the cameras focus on the reticle which, you did in step 4.

  1. Click Tools > Checkerboard Tool (stereo) to launch the Stereo Checkerboard Tool. Again, right-click on the Viewports and set the cameras (in the same order as in step 3). In each Viewport, you should see a rainbow-colored figure highlighting the corner points of the checkerboard:

checkerboard_rainbow

  1. Right-click each Viewport and select Parallax > Filters > Checkerboard (smooth). This switches to the more accurate (but slower) checkerboard detection algorithm for data collection.

  2. Click "Grab Corners" in the Stereo Checkerboard Tool to save the corner set. You should see the pose count in the "Save Corners" button increase to 1.

one_pose

  1. Move the checkerboard to a different position and orientation, ensuring that it is still in view and in focus in both cameras. If the corners cannot be detected, you will know because the rainbow highlight overlay will disappear. Once the corner detection overlay has settled (there is a 5-frame latency due to the smoothing algorithm) click "Grab corners" to grab another corner set.

newpose

  1. Repeat Step 9 until you've collected at least 25 corner sets from different orientations and sub-regions of the fields of view.

  2. Click "Save Corners" to save the corner set. A timestamped filename is suggested, but feel free to call it what you like.

  3. Close the Checkerboard Tool and return to the Main Window. Go to Tools > Calibration from Stereo Corners. This will launch a new window:

calfromstereo

  1. In the new window, click "Load" and select the corners file you saved in Step 11. Enter a name for the calibration you wish to generate (or leave the suggested, timestamped name). Leave the "provide intrinsics" box un-checked. Click "Generate Calibration". Close the "Calibration from Stereo Corners" tool. In the Main Window, you should now see your new calibration in the Calibration Panel:

newcal

  1. Remove the checkerboard from view and return to the targeting reticle.

NOTE: In this tutorial, I will re-use the checkerboard as my targeting reticle - the procedure is the same for both, but where I am using checkerboard corners as fiducial points, you will be using ticks on the reticle)

  1. From the Main Window, go to Tools > Rigid Body Transform Tool; this will open a new window:

rbt

  1. In the new window, enter the name of your global coordinate system ("reticle" or "headframe" or similar) in the "From Coordinate System" field. Enter "camera" (or similar) in the "To Coordinate System" field. In "Transform Name", pick a name for the new transform you are about to create - perhaps "reticle2camera" or similar. This is how your transform will later be identified in the Transform Panel.

  2. Make sure your calibration from Step 11 is selected in the Calibration Panel. In the Main Window Viewports, select a fiducial point in both views by left-clicking (you may wish to zoom in quite a bit - use the mousewheel for this). Click "Triangulate" in the Calibration Panel (or hotkey T on the keyboard). You should see the triangulated value of the fiducial point reported in the message log:

triangulated

  1. In the Rigid Body Tool again, enter the reticle coordinates of this fiducial point in the "From" coordinate fields. Then enter the triangulated coordinates in the "To" coordinate fields. Click the "right arrow" icon to register this data to your point set; it will disappear from the left panel and appear in the list on the right.

HINT: you can drag-and-drop the triangulated coordinates from the Calibration Panel into the Rigid Body Transform tool.

  1. Repeat step 18 for several fiducial points (at least 5).

  2. Once you have collected you correspondence points for the reticle-to-camera transform, it's a good idea to save them - in case you want to add to them later, or use them in other Analyses. Click "Save" and enter a filename; your correspondence points will be saved in CSV format.

  3. Now, click "Generate Transform" in the Rigid Body Transform tool. You should see your transform appear in the Transform Panel on the Main Window:

newtransform

Click "Save" to save the transform to disk.

  1. Now that you've generated the reticle-to-camera transform, the next step is to generate the camera-to-probe transform. This can be done manually using the Rigid Body Transform tool like we did before, but for the camera-to-probe transform, Parallax offers a special tool to streamline the process. Open this tool by clicking Tools > Transforms > Camera-to-Probe Transform Tool in the Main Window:

cpt

  1. In the Automation Panel of the Camera-to-Probe Transform tool, set the automation parameters:
  • Stage: Which stage to drive through the automation procedure.
  • Resolution: The number of lattice points per dimension of the cube. 2 will lead to 8 correspondence points, 3 will lead to 27, etc.
  • Extent: The side length, in microns, of the cubical lattice.
  • Origin: The center of the cubical lattice in stage coordinates. You can click "Set current position as origin" to set this from currently selected stage position.
  • Calibration: The calibration to be used when triangulating image points into object points for the correspondence.

When ready (make sure the probe is clear of obstacles!) click "Start Automation". The probe will move to the first correspondence point, and a message will appear in the message log, e.g:

Registration point 1 (of 8) reached: [5500.500000, 5499.500000, 1000.000000]

  1. In the Main Window, click-to-select the probe tip in both view ports in the Main Window, and then click "Register Points and Continue" in the Correspondence Panel of the Camera-to-Probe Transform Tool:

Screenshot from 2023-08-30 13-52-26

Screenshot from 2023-08-30 14-14-53

If you check the box labeled "Trigger from keyboard (C)", the routine will automatically register points and continue when the hot key "C" is pressed (requires keyboard focus on the Main Window).

Repeat until the automation procedure is complete (or, you can cancel prematurely).

  1. Once you've collected enough correspondence points (at least 8 is recommended), go to the "Generate Transform" panel of the Camera-to-Probe tool, fill in your transform metadata and click "Generate". The transform will be added to the the dropdown menu in the Transform Panel of the Main Window, and from there you can save the transform to disk.

  2. Now, go to the "From Composition" tab of the Rigid Body tool. Select your first transform "reticle2camera", and click Add. Then select your second transform "camera2probe" and click Add. Give the composited transform a name (perhaps "reticle2probe") and click Generate. The newly generated transform will appear in the Transform Panel. Click "Save" there to save it to disk.

  3. That's it! You've generated a coordinate transform from reticle coordinates to probe coordinates. To test it, make sure it is selected in the Transform Panel, and click Apply. This will open a new window that you can use to transform between the two systems.

apply_transform

On the left, enter the coordinates of a new fiducial point in the reticle system, and click "map forward" - the transformed coordinates (in the probe system) will appear on the left. You can manually drive the probe tip to this fiducial point and confirm that the motor coordinates agree. Once you are confident in the accuracy of the transform, you can simply drag-and-drop coordinates from the Apply Transform widget to the Targeting Dialog, and the probe will navigate to your reticle points automatically.

Clone this wiki locally