Annotator Model

[AmitMY]

Our end goal with this project is to transcribe long videos.
However, since we don't have long videos to train on, we can resort to creating an annotator model to annotate long videos using additional cues.

Annotator multitask model

SignWriting to spoken text and vice versa (https://github.com/sign-language-processing/signbank-plus/)
single-sign pose to SignWriting
multi-sign pose to SignWriting (any subset. 2, 3, 4....)
single-sign/multi-sign pose + the spoken language translation to SignWriting
pose to text

Inference: Annotate existing data

During inference, we run the segmentation model, and try to transcribe one sign at a time, with the spoken language context, without it, and as a multi-sign sequence, and ensemble the predictions.