Skip to content

Issues: mlcommons/modelbench

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

193 Open 178 Closed
193 Open 178 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Preserve existing run_data in standards file when regenerating the file
#805 opened Jan 18, 2025 by rogthefrog
@rogthefrog
update or kill PyPi releases
#804 opened Jan 17, 2025 by wpietri
Version number is off
#803 opened Jan 17, 2025 by wpietri
Start measuring how vendor-hosted model scores change over time
#802 opened Jan 16, 2025 by wpietri
Update supported Python versions
#800 opened Jan 15, 2025 by rogthefrog
@rogthefrog
1
Make modelbench anonymous (offline) practice runs use the 10% extract instead of the full practice set.
#798 opened Jan 15, 2025 by bollacker
Generated standards.json should preserve run info from previous runs
#797 opened Jan 15, 2025 by rogthefrog
UI to access benchmarks
#796 opened Jan 14, 2025 by wpietri
run the french models (first pass)
#795 opened Jan 14, 2025 by wpietri
1
For common models, document their frenchness
#794 opened Jan 14, 2025 by wpietri
french calibration
#793 opened Jan 14, 2025 by wpietri
@rogthefrog
1
Test switching from threads to processes
#792 opened Jan 14, 2025 by wpietri
Add pypi build test back to modelbench
#790 opened Jan 13, 2025 by rogthefrog
@rogthefrog
Improve modelbench CLI input validation + secret checks
#789 opened Jan 13, 2025 by bkorycki
Fix cache corruption bug
#785 opened Jan 9, 2025 by wpietri
Ensemble tests are registered in modelgauge
#782 opened Jan 8, 2025 by bkorycki
Improve how locales and prompt file name interact with each other
#781 opened Jan 7, 2025 by rogthefrog
@rogthefrog
1
Fix modelbench cache bug bug Something isn't working
#778 opened Jan 6, 2025 by bkorycki
fix the smoke tests
#777 opened Jan 2, 2025 by wpietri
Replace Locale object with just strings
#764 opened Dec 18, 2024 by rogthefrog
@rogthefrog
2
User bug: missing secrets for private modellab files when running public benchmark bug Something isn't working
#762 opened Dec 18, 2024 by bkorycki
2
Fix smoke test
#761 opened Dec 17, 2024 by rogthefrog
@rogthefrog
3
Make it obvious if annotator is not deployed successfully
#760 opened Dec 17, 2024 by rogthefrog
1
Rerun 3 SUTs with French prompts
#759 opened Dec 17, 2024 by rogthefrog
@rogthefrog
2
Get rid of SUT wrapper
#753 opened Dec 16, 2024 by bkorycki
ProTip! Type g p on any issue or pull request to go back to the pull request listing page.