Flow Elaboration & PSL Lexicon

[imsamimalik]

@AmitMY I have a little confusion understanding the flow of this process.

Why do we need to convert each text into the gloss, because, at the next step, you are using the gloss to find its relevant video? Can't we just skip the gloss part and just use text to find its video and proceed forward?

And what lexicon are you using for PSL(Pakistan Sign Language)?

Thanks!

[imsamimalik]

And is it powering the https://sign.mt/ website? When I explored this repo, many implementations were missing.

[AmitMY]

We have a paper describing the flow: https://arxiv.org/pdf/2305.17714.pdf

As for your question, we do search by words at first, but in reality, words are conjugated into different forms, in german for example, the verb changes depending on the subject. i "esse", you "isst", they "essen" - and the dictionary will likely only include the base form, "essen". glossifying (or in our default implementation, lemmatization) strips away this variety.

As for PSL - we use spreadthesign in https://sign.mt but ideally we would need to move away from it.

The "implementations" missing that you refer to - I understand those as missing lexicons, not code, since what is running in sign.mt is this repository, with a different lexicon. There are many lexicons one could add (for example sign-language-processing/datasets#32) but it is not our focus, and should be simple for end users to add

[imsamimalik]

Oh okay, I got it. I noticed the NotImplemented placeholder in the pose-to-video module.

[imsamimalik]

@AmitMY Sorry to disturb you again.

How do you extract a pose file from a video?

I have tried a few methods but they do not fulfill my needs.

• I have tried the original OpenPose Python Library. But its model URL has not been working(the link where models are hosted is down). So eventually the absence of models makes it useless.
• I have tried ControlNet. It can give me individual frames image files with pose detected but I have no clue how to convert it to a pose or JSON file.

Any help would be appreciated.

[AmitMY]

Use MediaPipe holistic
easiest way imo is

pip install git+https://github.com/sign-language-processing/transcription
pip install mediapipe 

video_to_pose -i sign.mp4 --format mediapipe -o sign.pose