Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Faster strm exports #320

Open
wants to merge 1 commit into
base: matrix
Choose a base branch
from
Open

Faster strm exports #320

wants to merge 1 commit into from

Conversation

0o120
Copy link

@0o120 0o120 commented Feb 2, 2023

Google Drive Test Directory:

Export Directory:
    /Movies/

Contents:
    /Movies/Movie1/Movie1.mp4
    /Movies/Movie2/Movie2.mp4
    /Movies/Movie3/Movie3.mp4
    ... 2,292 total movies

Before Changes
Google Drive API Requests: 2317
Export Duration: 15 minutes and 40 seconds (940 seconds)

After Changes
Google Drive API Requests: 28
Export Duration: 4 minutes and 26 seconds (266 seconds)

Could be faster, but I think it's slowed down with the way clouddrive is handling each google drive item. This is what I could do with minimal changes.

It fetches all the files\folders in the directory with minimal api requests and then just feeds it back to clouddrive with the providers instance cache.

@0o120
Copy link
Author

0o120 commented Feb 2, 2023

Was able to get that same 266 second export down to 47 seconds by disabling the export_manager database access inside the while loop located inside clouddrive.common.service.export.ExportService.process_pending_changes and saving the items after the loop.

You can see in these changes:
0o120/script.module.clouddrive.common-1@c5a5b7b

I'm assuming export_manager.save_pending_changes() is used to save progress in the event of an interruption and not exactly necessary? Maybe adding a config option to disable save_pending_changes inside that loop in exchange for performance gain?

I'm up for adding a config option. Some thoughts on this would be great :)

@cguZZman
Copy link
Owner

Thanks for the improvement. It's been a long time and this project is been in KTLO mode for me. But I will merge your change if you confirm it's been working fine for you.
Also, saving the progress is necessary for large catalogs. If you submit your other change with a config option, I could merge it too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants