-
Notifications
You must be signed in to change notification settings - Fork 489
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
implement ContentServerDataStore for azure blob storage too #1234
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you very much for this PR, it's greatly appreciated! I had a brief first look and left a few comments. Let me know what you think!
pkg/azurestore/azureservice.go
Outdated
if err != nil { | ||
return err | ||
} | ||
resp, err := blockBlob.BlobClient.DownloadStream(ctx, downloadOptions) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we also forward headers like If-Match to Azure for supporting conditional requests? This is done by the s3store as well:
tusd/pkg/s3store/serve_content.go
Lines 31 to 48 in a50bc42
if val := r.Header.Get("If-Match"); val != "" { | |
input.IfMatch = aws.String(val) | |
} | |
if val := r.Header.Get("If-None-Match"); val != "" { | |
input.IfNoneMatch = aws.String(val) | |
} | |
if val := r.Header.Get("If-Modified-Since"); val != "" { | |
t, err := http.ParseTime(val) | |
if err == nil { | |
input.IfModifiedSince = aws.Time(t) | |
} | |
} | |
if val := r.Header.Get("If-Unmodified-Since"); val != "" { | |
t, err := http.ParseTime(val) | |
if err == nil { | |
input.IfUnmodifiedSince = aws.Time(t) | |
} | |
} |
Or does Azure Blob Storage not support conditional requests?
pkg/azurestore/azureservice.go
Outdated
} else if resp.ContentLength != nil && *resp.ContentLength == 0 { | ||
statusCode = http.StatusNoContent | ||
} | ||
w.WriteHeader(statusCode) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we forward response headers like Content-Type from Azure's response? Similar to how its done in the s3store:
tusd/pkg/s3store/serve_content.go
Lines 95 to 128 in a50bc42
// Add Accept-Ranges,Content-*, Cache-Control, ETag, Expires, Last-Modified headers if present in S3 response | |
if result.AcceptRanges != nil { | |
w.Header().Set("Accept-Ranges", *result.AcceptRanges) | |
} | |
if result.ContentDisposition != nil { | |
w.Header().Set("Content-Disposition", *result.ContentDisposition) | |
} | |
if result.ContentEncoding != nil { | |
w.Header().Set("Content-Encoding", *result.ContentEncoding) | |
} | |
if result.ContentLanguage != nil { | |
w.Header().Set("Content-Language", *result.ContentLanguage) | |
} | |
if result.ContentLength != nil { | |
w.Header().Set("Content-Length", strconv.FormatInt(*result.ContentLength, 10)) | |
} | |
if result.ContentRange != nil { | |
w.Header().Set("Content-Range", *result.ContentRange) | |
} | |
if result.ContentType != nil { | |
w.Header().Set("Content-Type", *result.ContentType) | |
} | |
if result.CacheControl != nil { | |
w.Header().Set("Cache-Control", *result.CacheControl) | |
} | |
if result.ETag != nil { | |
w.Header().Set("ETag", *result.ETag) | |
} | |
if result.ExpiresString != nil { | |
w.Header().Set("Expires", *result.ExpiresString) | |
} | |
if result.LastModified != nil { | |
w.Header().Set("Last-Modified", result.LastModified.Format(http.TimeFormat)) | |
} |
pkg/azurestore/azureservice.go
Outdated
return nil, fmt.Errorf("invalid offset in Range header") | ||
} | ||
|
||
count, err := strconv.ParseInt(rangeParts[1], 10, 64) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You mentioned that "Incomplete range requests (Range: bytes=100- ) are not supported". From the SDK's documentation, it seems to be possible to let the count be zero and get the entire resource starting at the offset:
An HTTPRange which has an offset and zero value count indicates from the offset to the resource's end.
Could you implement support for this?
…more headers and Range without rangeEnd
yeah, as suggested, azure supports all those headers too. please have a look, i gave it a new try. tests with azurite do work.. do you see anything else missing? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you very much for the quick updates! The handler is the request headers looks much better yet. Could you also have a look at the failing tests at https://github.com/tus/tusd/actions/runs/12736636713/job/35496766413?pr=1234? Talking about tests, it would be great to have some tests for ServeContent in azurestore as well. For inspiration you can use s3store's tests: https://github.com/tus/tusd/blob/main/pkg/s3store/serve_content_test.go
@@ -67,6 +68,8 @@ type AzBlob interface { | |||
Upload(ctx context.Context, body io.ReadSeeker) error | |||
// Download returns a readcloser to download the contents of the blob | |||
Download(ctx context.Context) (io.ReadCloser, error) | |||
// Serves the contents of the blob directly handling special HTTP headers like Range, if set | |||
ServeContent(ctx context.Context, w http.ResponseWriter, r *http.Request) error |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you please run go fmt
to address the formatting inconsistencies?
@@ -253,6 +281,11 @@ func (infoBlob *InfoBlob) Download(ctx context.Context) (io.ReadCloser, error) { | |||
return resp.Body, nil | |||
} | |||
|
|||
// ServeContent is not needed for infoBlob | |||
func (infoBlob *InfoBlob) ServeContent(ctx context.Context, w http.ResponseWriter, r *http.Request) error { | |||
return nil |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
return nil | |
return errors.New("azurestore: ServeContent is not implemented for InfoBlob") |
Please return an error here so we notice it early if this method is accidentally called.
} else if resp.ContentLength != nil && *resp.ContentLength == 0 { | ||
statusCode = http.StatusNoContent | ||
} | ||
w.WriteHeader(statusCode) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we forward response headers like Content-Type from Azure's response? Similar to how its done in the s3store:
tusd/pkg/s3store/serve_content.go
Lines 95 to 128 in a50bc42
// Add Accept-Ranges,Content-*, Cache-Control, ETag, Expires, Last-Modified headers if present in S3 response | |
if result.AcceptRanges != nil { | |
w.Header().Set("Accept-Ranges", *result.AcceptRanges) | |
} | |
if result.ContentDisposition != nil { | |
w.Header().Set("Content-Disposition", *result.ContentDisposition) | |
} | |
if result.ContentEncoding != nil { | |
w.Header().Set("Content-Encoding", *result.ContentEncoding) | |
} | |
if result.ContentLanguage != nil { | |
w.Header().Set("Content-Language", *result.ContentLanguage) | |
} | |
if result.ContentLength != nil { | |
w.Header().Set("Content-Length", strconv.FormatInt(*result.ContentLength, 10)) | |
} | |
if result.ContentRange != nil { | |
w.Header().Set("Content-Range", *result.ContentRange) | |
} | |
if result.ContentType != nil { | |
w.Header().Set("Content-Type", *result.ContentType) | |
} | |
if result.CacheControl != nil { | |
w.Header().Set("Cache-Control", *result.CacheControl) | |
} | |
if result.ETag != nil { | |
w.Header().Set("ETag", *result.ETag) | |
} | |
if result.ExpiresString != nil { | |
w.Header().Set("Expires", *result.ExpiresString) | |
} | |
if result.LastModified != nil { | |
w.Header().Set("Last-Modified", result.LastModified.Format(http.TimeFormat)) | |
} |
Hi & Happy new year
Implement ContentServerDataStore for azure blob storage too, tests with azurite+azure blob are ok.
Incomplete range requests (Range: bytes=100- ) are not supported, hope that's ok
Regards