Add support for DLS in Dropbox #1839
Conversation
connectors/sources/dropbox.py
Outdated
| else: | ||
| return document, None | ||
|
|
||
| if filtering and filtering.has_advanced_rules(): | ||
| if self._dls_enabled(): | ||
| account_id, member_id = await self.get_account_details() |
There was a problem hiding this comment.
It's better to decouple this block of code into a separate method.
|
Will update the PR soon with suggested change:https://crestdatasystems.slack.com/archives/C02AV5C8ZDE/p1698684375821709?thread_ts=1698657367.352959&cid=C02AV5C8ZDE |
| @@ -35,7 +40,10 @@ | |||
| DEFAULT_RETRY_AFTER = 300 # seconds | |||
| RETRY_INTERVAL = 2 | |||
| MAX_CONCURRENT_DOWNLOADS = 100 | |||
| REQUEST_BATCH_SIZE = 50 | |||
There was a problem hiding this comment.
Could you please explain how you came up with this number?
There was a problem hiding this comment.
max batch size we can pass is 100 as per documentation but as we are storing the complete document with the file id in dictionary on line 1137 so it's throwing Queue Full error while running the connector. To fix this issue we tried to pass less number document in batch and it's working fine with REQUEST_BATCH_SIZE 50
…ython into dropbox-dls
💔 Failed to create backport PR(s)The backport operation could not be completed due to the following error: The backport PRs will be merged automatically after passing CI. To backport manually run: |
Part of https://github.com/elastic/connectors/issues/1819###
Checklists
Pre-Review Checklist
v7.13.2,v7.14.0,v8.0.0)