Sleap generates dozens/hundres of unique instances even though only trained on 2 instances

[catubc]

Hello
We are long time users of sleap. We are currently using sleap to track the location of 2 physical wooden blocks (that have slightly different colors/features) inside an animal cage.

Sleap does a great job in some videos, but in others it generates dozens, sometimes hundreds of tracks as in the image attached.

Is there something easy we can do to avoid this? I thought sleap would only ever generate max of n tracks based on training data.

Thanks for the help.
@catubc

[roomrys]

Hi @catubc,

Info you didn't ask for that may or may not help

SLEAP currently has 2 methods for tracking: ID-based tracking using a neural network (which requires annotating the tracks for each labeled block) and temporal-based tracking (which just uses a distance or overlap metric between blocks in sequential frames).

For ID-based tracking I would expect there to only be however many tracks you have annotated. For temporal-based tracking, there is now an options to set a limit of how many tracks are created (an instead recycle unused tracks). This option is here:

or for the sleap-track cli:

  --tracking.max_tracking TRACKING.MAX_TRACKING
                        If true then the tracker will cap the max number of tracks. (default: False)
  --tracking.max_tracks TRACKING.MAX_TRACKS
                        Maximum number of tracks to be tracked by the tracker. (default: None)

.

A question for you

I am a bit suprised by these extraneous track generations since... well... we're tracking blocks here (that don't move much and should be trivial to temporally-track). What are your tracking settings (I am assuming you are using temporal-based tracking)? A screenshot of the GUI should work. Another piece that would help is a screenshot (or a few) of the frames where these switches happen. The most often case of a switch is when the item being tracked is occluded - all it takes is a single frame, then if "Connect Single Track Breaks" is not set to True, a new track will be generated for the next frame that the item is found again.

Suspicions:
I am suspicious about #1447 which implemented the max tracking limit (and think it may have lead to some breaks elsewhere - e.x. #1583).

Thanks,
Liezl

[lisadiez]

Hi Liezl :)

Thank you for the quick response - as always!

We trained and predicted on these back in March, when I think you didn't have the track limit option. I'll definitely try that.

Re: Your questions:

• Tracking settings (as suggested in #1224)
  sleap-track <data-path> \ -m <model> \ --tracking.tracker simple \ --tracking.target_instance_count 2 \ --tracking.pre_cull_to_target 1 \ --tracking.pre_cull_iou_threshold 0.8 \ --tracking.post_connect_single_breaks 1

• Screenshots. You're right - what usually happens is a gerbil walks over the blocks and then the switch occurs (see video)

We'll keep you updated when we try the track limit, but if you have any more suggestions let us know!

Best,
Lisa

[roomrys]

Hi @lisadiez,

Ah it looks like the tracking settings in #1224 are the same as I would have suggested below... and the switches to new tracks do indeed occur when both blocks are lost simultaneously.

---

Since there are only two objects being tracked (making it more likely that only one goes missing at a time) - on the older GUI, you can check "Cull to Target Instance Count" and then set the "Target Number of Instances Per Frame" to 2. Additionally, what will really help here is to check "Connect Single Track Breaks".

We need to set the "Target Number of Instances Per Frame" because the "Connect Single Track Breaks" logic requires it:

sleap/sleap/nn/tracking.py

Lines 801 to 802 in cb82d36

	elif self.target_instance_count and self.post_connect_single_breaks:
	connect_single_track_breaks(frames, self.target_instance_count)

Thanks,
Liezl

[talmo]

Closing -- as of v1.3.3 we now have track-local queues to enforce the maximum number of tracks globally.