Calibration result view goes blank if just one point's calibration fails

[datalowe]

Steps to reproduce:

1. Run one of the demo's calibration procedure.
2. (as the participant) Close your eyes or look away entirely from the screen for one of the calibration points, and focus on the remaining calibration points.
   i. It's possible that it has to be the last point that you look away for, but I think that the issue appears regardless of which point you close your eyes for.
3. Proceed to the calibration results view.
   The calibration results view should now be entirely 'blank'.

What I expected to happen:

Rather than getting an entirely 'blank' calibration results view, I'd expect that only the one point/area for which you closed your eyes would be blank, while other calibration points would have corresponding red/green lines indicating how far participant gaze was from the target.

Cause of issue

In the package's init.py file inside of the _show_calibration_result method, there is the following check:

if self.calibration_result.status == tr.CALIBRATION_STATUS_FAILURE:
    # computeCalibration failed.
    pass

This status attribute is assigned its value through the return value of a tobii_research function call. Tobii's documentation here is somewhat opaque, but as far as I could understand, this value is set to the constant CALIBRATION_STATUS_FAILURE as soon as any point's calibration data collection fails (regardless of whether or not collection was successful for other points).

Suggested solution

Simply remove the self.calibration_result.status == tr.CALIBRATION_STATUS_FAILURE check. As far as I can tell, and from testing, this shouldn't cause any issues. The _show_calibration_result code relies on self.calibration_result.calibration_points, which as far as I can tell only includes points for which calibration data collection was 'successful' (ie participant gaze could be captured at all). I tried this in a branch and with the modification, the results presentation behavior is as I would expect.

In fact, now that I look at it again, it's probably possible to drop the if len(self.calibration_result.calibration_points) == 0: check as well, since if the length is indeed 0 then for this_point in self.calibration_result.calibration_points: loop won't do any iterations anyway. I unfortunately can't test this myself since I don't have access to an eyetracker now, but it might be worth considering, for cleaning the code.

If the solution seems appropriate, I can do a PR.

[yh-luo]

Thanks for the report! Unfortunately, the solution is not viable.
From the documentation in tobii_research.ScreenBasedCalibration.compute_and_apply:

    def compute_and_apply(self):
        '''Uses the data in the temporary buffer and tries to compute calibration parameters.

        If the call is successful, the data is copied from the temporary buffer to the active buffer.
        If there is insufficient data to compute a new calibration or if the collected data is not
        good enough then an exception will be raised.

        See @ref find_all_eyetrackers or EyeTracker.__init__ on how to create an EyeTracker object.
        <CodeExample>calibration.py</CodeExample>
        Raises:
        EyeTrackerConnectionFailedError
        EyeTrackerFeatureNotSupportedError
        EyeTrackerInvalidOperationError
        EyeTrackerLicenseError
        EyeTrackerInternalError

        Returns:
        A CalibrationResult object.
        '''
        interop_result = interop.screen_based_calibration_compute_and_apply(self.__core_eyetracker)

        status = _calibration_status[interop_result[0]]
        if (status != CALIBRATION_STATUS_SUCCESS):
            return CalibrationResult(status, ())
        # the remaining code ...

The calibration result will be used by the eye tracker if the call is successful, otherwise CALIBRATION_STATUS_FAILURE will be returned. That means the calibration result will NOT be used by the eye tracker if it is not good enough.

_show_calibration_result shows nothing when the calibration is not successful - since no calibration is done!
We can of course show the calibration samples, but that will not be meaningful when those samples are not used to calibrate the eye tracker.

I agree that the if-else check of len(self.calibration_result.calibration_points) == 0 should be dropped. However, the modification needs to be tested as you said. Would you be willing to create a PR for this? Or I would pick up the issue next week.

[yh-luo]

@datalowe I planned to clean up the code on the weekends (maint branch).
If you're occupied by other things please let me know, I will work on the problems you mentioned.
As usual, no pressure and thank you for bringing it up!

[datalowe]

Hi, sorry for taking so long to respond.

If I do a calibration and data are captured for points 1, 2, and 4 (while points 3 and 5 fail entirely), the calibration as a whole is unsuccesful and not applied. However, if I then recalibrate points 3 and 5 successfully, I thought the calibration would be successful and apllied. Am I mistaken here?

Since I thought the successful data capture for a subset of calibration points could be used together with additional later (recalibration) capture data, I figured the experimenter should be given feedback about this. Doing so would prevent unnecessarily recalibrating at points for which there are already valid data. Hence I suggested the change proposed in this issue.

But of course, this is all based on my understanding that "partial" calibration data can be "filled out" by a partial recalibration procedure. I don't have access to an eyetracker now, so I can't test this unfortunately.

[yh-luo]

However, if I then recalibrate points 3 and 5 successfully, I thought the calibration would be successful and apllied. Am I mistaken here?

You are correct about that, but that's a different situation. When we conduct calibration, there are many possible situations:

1. The calibration samples are not good enough, Tobii SDK returns CALIBRATION_STATUS_FAILURE. The eye tracker is not calibrated.
2. The calibration samples are ok for calibration, Tobii SDK returns CALIBRATION_STATUS_SUCCESS. The eye tracker is calibrated using the samples._show_calibration_result thus displays the calibration samples and the calibration points. Users can choose to recalibrate some unsatisfying points, these calibration samples will be discarded (self.calibration.discard_data) and replaced with new calibration samples.
3. The calibration samples are ok for calibration and users accept the result.

The issue you mentioned is situation 1 and the following scenario is situation 2. When CALIBRATION_STATUS_FAILURE is returned, the eye tracker is not calibrated at all.
It is possible that, as you said, in situation 1 there are valid samples for a part of the calibration points. But we don't know how Tobii SDK handles those samples when the calibration is not successful. _collect_calibration_data pass the data to tobii_research.ScreenBasedCalibration.collect_data, which is not open-source. It is possible that those data are discarded or kept for future calibration, but we can't be sure since the documentation does not clarify.

As a side note, tobii_research.ScreenBasedCalibration.collect_data stated that

The argument used is the point the calibration user is assumed to be looking at and is given in the active display area coordinate system.

When the user violates the assumption, we don't know how Tobii SDK handles those data. It would be too risky to assume that Tobii will keep those data for future calibration. That's why I don't want to show anything when the calibration is not ok (situation 1).

[datalowe]

When the user violates the assumption, we don't know how Tobii SDK handles those data. It would be too risky to assume that Tobii will keep those data for future calibration. That's why I don't want to show anything when the calibration is not ok (situation 1).

I see, it's unfortunate that Tobii's documentation isn't very clear about this. Would you agree though that it should be safe to assume that whenever 'CALIBRATION_STATUS_SUCCESS' is returned, the calibration really has been applied? If so, it should be possible to add a separate indicator eg at the bottom of the calibration results screen that informs the user about the 'calibration status'. It could read:

• 'Data collected for all calibration points (evaluate data quality above)' (green text color) if CALIBRATION_STATUS_SUCCESS was returned
• 'Data collection FAILED for one or more calibration points, calibration NOT applied' (red text color) if CALIBRATION_STATUS_FAILURE was returned

Then, the flow as seen from the experimenter's point of view could go like this:

1. Runs calibration.
2. (in calibration results view) Sees red/green lines going out from only 3 points, and 'Data collection FAILED for one or more calibration points, calibration NOT applied' at bottom of screen.
3. Selects (at least) the points which have no outgoing red/green lines.
4. Reruns calibration for selected points.
5. (in calibration results view) Sees red/green lines going out from all points, and 'Data collected for all calibration points' at bottom of screen.

If it would turn out that Tobii does something really weird and it's possible that in step 5 there would be red/green lines going out from all points (ie data have been collected for all points) but CALIBRATION_STATUS_FAILURE is still returned (ie calibration was not applied), then the user could still tell from the error message at the bottom that the calibration failed anyway. They would presumably then attempt to rerun the calibration entirely.

Based on my and colleagues' (they are the ones who have the eyetracker now) testing, the eyetracker, or tobii_research's objects, do 'remember' the calibration data from a partially successful calibration. We've observed the behavior of 'additional' points having red/green lines after running a recalibration only with the 'failed' points, like in steps 4 and 5 in the example. It seems unlikely to me that the Tobii software writers would have made the eyetracker not apply a calibration even when it reports successfully captured calibration data for all points, but I might be missing something.

[yh-luo]

Would you agree though that it should be safe to assume that whenever 'CALIBRATION_STATUS_SUCCESS' is returned, the calibration really has been applied?

At least that's what the documentation states, I believe so.

The proposal sounds reasonable to me. I would see what I can do recently (I don't have access to the eye tracker at the moment. It may take some time to make arrangement.)

In my experience, CALIBRATION_STATUS_FAILURE is rarely returned if the experimenter carefully collect the calibration samples. Even in that situation, recalibration should work well as you said. Let's keep this issue (improvement) open until it's solved!