Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature - Multipart Uploads #324

Merged
merged 13 commits into from
Jan 28, 2025
Merged

Feature - Multipart Uploads #324

merged 13 commits into from
Jan 28, 2025

Conversation

shincap8
Copy link
Contributor

Context

  • There are files big enough that our backend can't post directly to s3 so we are implementing multipart upload

Changes Made

  1. Include endpoint for generating presigned URLS.
  • This would allow the frontEnd to upload the file in chunks.
  1. Include endpoint for completing the upload.
  • This would allow us to complete the multipart upload, generate the model in the DB and call Runpod.
  1. Include endpoint to abort multipart.
  • This would help us abort the uploading when theres is an error.

shincap8 and others added 13 commits January 16, 2025 19:25
@shincap8
Update model methods to run background task when creating the endpoin…
04cd514 
…t and creating score endpoint for runpod to use
@shincap8
Handle error from Runpod, URL inference from yaml file
d86ae16
@shincap8 @remg1997
Update backend/app/api/endpoints/base/score.py
8794693 
Co-authored-by: Rafael Mosquera <[email protected]>
@shincap8
Rename method
fc08634
@shincap8
Merge with cloud branch
aa0f022
@shincap8
rename endpoint
971b986
@shincap8
Print background tasks and correct typo
476d4e4
@shincap8
resolve merge conflict
9ed5f20
@shincap8
Merge branch 'main' of https://github.com/mlcommons/dynabench
60f6db6
@shincap8
Merge branch 'main' of https://github.com/mlcommons/dynabench into Fe…
7d01138 
…ature/multipart-uploads
@shincap8
New methods for multipart upload
80bd023
@shincap8
Merge branch 'main' of https://github.com/mlcommons/dynabench into Fe…
dc1c005 
…ature/multipart-uploads
@shincap8
Backend logic for multipart, 3 new endpoints, one for creating the si…
658a323 
…gned urls other for completing the multipart upload and the third for aborting
@shincap8 shincap8 requested a review from a team as a code owner January 28, 2025 00:21
@github-actions GitHub Actions
Copy link

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@shincap8 shincap8 merged commit 30126a2 into main Jan 28, 2025
13 checks passed
@shincap8 shincap8 deleted the Feature/multipart-uploads branch January 28, 2025 14:44
@github-actions github-actions bot locked and limited conversation to collaborators Jan 28, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@remg1997 remg1997 remg1997 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@shincap8 @remg1997