Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add MLQA #2622

Merged
merged 7 commits into from
Jan 15, 2025
Merged

Add MLQA #2622

merged 7 commits into from
Jan 15, 2025

Conversation

KahnSvaer
Copy link
Contributor

@KahnSvaer KahnSvaer commented Jan 13, 2025

  • add mlqa_common_yaml

  • add 49 tests of mlqa family

  • update tasks/README.md


resolved: #192

* add mlqa_common_yaml

* add 49 tests of mlqa family

* update tasks/README.md

---------
@KahnSvaer
Copy link
Contributor Author

I am not sure about the doc_to_text prompt that I wrote. I have checked that the code works with dummy model and that the process_result function is working as intended. I have also checked that it works with pythia model though I was not able to completely evaluate it on the dataset due to gpu constraints. Most of the code in tasks/mlqa/utils.py is directly taken from the main repo: https://github.com/facebookresearch/MLQA

@KahnSvaer
Copy link
Contributor Author

Found the error sending the updated commit as soon as possible

@baberabb
Copy link
Contributor

baberabb commented Jan 15, 2025

Hi @KahnSvaer, this is great! small nit: the test failing as it's trying to parse mlqa_common_yaml.yaml. just need to remove the file extension and should be good to go.

Have you come across any papers which test these tasks on any transformer models? I can try to reproduce. But no worries if not.

@KahnSvaer
Copy link
Contributor Author

KahnSvaer commented Jan 15, 2025

@baberabb The original paper for this benchmark https://arxiv.org/pdf/1910.07475 shows some tests on transformer models like M-Bert and XLM.

@baberabb
Copy link
Contributor

LGTM! thanks for the PR!

@baberabb baberabb merged commit e86cece into EleutherAI:main Jan 15, 2025
8 checks passed
@KahnSvaer KahnSvaer deleted the mlqa branch January 16, 2025 04:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement MLQA
2 participants