-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
✨feat(source-microsoft-sharepoint): add file transfer mode #54140
✨feat(source-microsoft-sharepoint): add file transfer mode #54140
Conversation
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
@@ -271,7 +271,7 @@ | |||
"required": ["filetype"] | |||
}, | |||
{ | |||
"title": "Document File Type Format (Experimental)", | |||
"title": "Unstructured Document Format", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This title was updated at some moment between CDK versions.
/approve-regression-tests
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks like a great start. Knowing we will update a few more things in sharepoint, I think it's fine to move forward.
Approving pending nits on deps and base image updates, and on green CI.
airbyte-integrations/connectors/source-microsoft-sharepoint/metadata.yaml
Outdated
Show resolved
Hide resolved
airbyte-integrations/connectors/source-microsoft-sharepoint/pyproject.toml
Outdated
Show resolved
Hide resolved
…11, pytest ^8, requests-mock ^1.12.1
…add-file-transfer
/approve-regression-tests
|
What
Implement file-transfer feature for source-microsoft-sharepoint
resolves: [source-microsoft-sharepoint]: Implement file transfer
How
It updated the spec to show the option that leverages cdk logic to handle the new record type and add file_size() and get_file() to the stream reader.
Review guide
airbyte-integrations/connectors/source-microsoft-sharepoint/source_microsoft_sharepoint/stream_reader.py
: new get_file method to download file from sharepoint in chunks. Also override of _get_file_transfer_paths to get correct path from file.uri. Also, use the file_size method to get the file size as we limit 1.5 GB downloads.others
: unit tests and updates on spec, documentation and connector metadata.Tets environment on Cloud
You can find some connection testing this feature here.
User Impact
If they just need to move files as they are, this will greatly improve users' experience as sync will be quite fast.
Can this PR be safely reverted and rolled back?