Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove file-handle from object store GET operations #7154

Open
crepererum opened this issue Feb 19, 2025 · 0 comments
Open

Remove file-handle from object store GET operations #7154

crepererum opened this issue Feb 19, 2025 · 0 comments
Labels
object-store Object Store Interface

Comments

@crepererum
Copy link
Contributor

What

Remove GetResultPayload::File (and the entire enum).

Why

The file handle has several drawbacks as it does NOT specify the following properties:

  • mode: Is the file opened read-only or does it also have write access (which would be bad)?
  • seek position: At which position is the current file "read head"? Are users expected to seek to the file start if they want to read the entire file?
  • direct IO: Is the file opened using direct IO? (in which case the user would have to read in aligned blocks)
  • exclusive access: Does the API user have exclusive access to the said file handle (and hence can use seek-based operations) or not (in which case only non-seeking operations like read_at would be required)

At the same time I suspect that most users copy the file data into a buffer anyways and don't use any kernel-based IO operations like sendfile.

All object store users and integrating libraries potentially have to deal with streams AND file handles.

How

Remove the entire GetResultPayload enum and just use streams.

@crepererum crepererum added the object-store Object Store Interface label Feb 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
object-store Object Store Interface
Projects
None yet
Development

No branches or pull requests

1 participant