Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a section on back-translation best practice, wrt to evaluation #83

Open
cleong110 opened this issue Jun 18, 2024 · 3 comments
Open

Comments

@cleong110
Copy link
Contributor

          possible write what `ham2pose` suggests, here.

maybe also need a note about back translation: that people use it (progressive transformers, sign llm), but the outputs are incoherent. this is because people train the backtranslation models on the translation model outputs, and not independently, as one should.

Originally posted by @AmitMY in #77 (comment)

@cleong110
Copy link
Contributor Author

cleong110 commented Jun 18, 2024

Possibly relevant here:

@cleong110
Copy link
Contributor Author

@inproceedings{huangFastHighQualitySign2021,
  title = {Towards {{Fast}} and {{High-Quality Sign Language Production}}},
  booktitle = {Proceedings of the 29th {{ACM International Conference}} on {{Multimedia}}},
  author = {Huang, Wencan and Pan, Wenwen and Zhao, Zhou and Tian, Qi},
  year = {2021},
  month = oct,
  series = {{{MM}} '21},
  pages = {3172--3181},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  doi = {10.1145/3474085.3475463},
  url = {https://doi.org/10.1145/3474085.3475463},
  urldate = {2024-06-19},
  isbn = {978-1-4503-8651-7}
}

has this to say:

image

@cleong110
Copy link
Contributor Author

Which seems to suggest that the method is:

  1. Generate poses
  2. run the code from "Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation" on it
  3. See the BLEU score, etc.

With the theory being that as you generate better, the SLT should get more accurate.

They don't train a backtranslation model themselves, that's independent

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant