Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/new metrics #322

Merged
merged 17 commits into from
Jan 22, 2025
Merged

Feature/new metrics #322

merged 17 commits into from
Jan 22, 2025

Conversation

shincap8
Copy link
Contributor

@shincap8 shincap8 commented Jan 22, 2025 •
edited
Loading

Context

  • MLSupeerb Challenge leaderboard metrics (Standard_Accuracy, CER, STD_CER, CER_15_WORSE, Dialect Accuracy, Dialect CER) weren't being displayed.

Changes Made

  1. Include all these metrics in both backends.
  • This would allow the user to see the leader board with the correct metrics.
image

shincap8 and others added 16 commits January 16, 2025 19:25
@shincap8
Update model methods to run background task when creating the endpoin…
04cd514 
…t and creating score endpoint for runpod to use
@shincap8
Handle error from Runpod, URL inference from yaml file
d86ae16
@shincap8 @remg1997
Update backend/app/api/endpoints/base/score.py
8794693 
Co-authored-by: Rafael Mosquera <[email protected]>
@shincap8
Rename method
fc08634
@shincap8
Merge with cloud branch
aa0f022
@shincap8
rename endpoint
971b986
@shincap8
Print background tasks and correct typo
476d4e4
@shincap8
resolve merge conflict
9ed5f20
@shincap8
Merge branch 'main' of https://github.com/mlcommons/dynabench into Fe…
326acbb 
…ature/RunPod
@shincap8
Correct typos and correct schemas
68f126e
@shincap8
model uid
c2638b9
@shincap8
merge with main
605d8fb
@shincap8
remove c
0d74498
@shincap8
Merge branch 'main' of https://github.com/mlcommons/dynabench into Fe…
9c6f088 
…ature/RunPod
@shincap8
Merge branch 'main' of https://github.com/mlcommons/dynabench into Fe…
ac0b90b 
…ature/RunPod
@shincap8
New metrics for MLSuperb challenge
bbb2695
@shincap8 shincap8 requested a review from a team as a code owner January 22, 2025 22:19
@github-actions GitHub Actions
Copy link

github-actions bot commented Jan 22, 2025 •
edited
Loading

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

remg1997
remg1997 previously approved these changes Jan 22, 2025
View reviewed changes
@shincap8 shincap8 merged commit 275c06d into main Jan 22, 2025
13 checks passed
@shincap8 shincap8 deleted the Feature/new_metrics branch January 22, 2025 22:52
@github-actions github-actions bot locked and limited conversation to collaborators Jan 22, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@remg1997 remg1997 remg1997 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@shincap8 @remg1997