Feature: allow folders to be continued more usefully #288
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
achadwick
force-pushed
the
continue-folder
branch
from
5e5fc56
to
7245742
Compare
|
Apologies for the forced push. @wkentaro, any chance you could merge these now, or is there anything more you'd like to see? I've adapted the patch to your move of Happy to review and update it to match your requirements. |
|
Just a random feedback: this feature is very useful. I'll install the fork one to have this. |
Document existing gdown behaviour.
Allow folder mode to pass through the resume flag. This still makes some unnecessary requests, however. Duplicates wkentaro#275, but also updates cli.py appropriately.
As an optimization, if the target file exists and gdown is in resume mode, don’t even try to transfer the file again. Obviously, this only makes sense when the atomic tempfile+rename pattern is in use, meaning only when output is to a path. This avoids unnecessary requests. Even a ranged GET with 0 bytes left counts against Google’s download limits, and it’s just faster. The --continue/resume flag is handled at both the file level and the folder level, to avoid even more requests. This noticeably improves performance and when mirroring folders with lots of files, when attempting to resume the transfer at a later time. It also allows better resumption of folder downloads that got interrupted. Update the function docs. Update help text for --continue to cover the new behaviour.
This is nicer for a lot of Linux thumbnailers, which will now avoid generating preview thumbnails for the partially transferred files.
Now that gdown handles fully transferred files properly, by ignoring them, the tempfile finder loop should only select files ending with the right suffix (currently “(random).part”). This avoids common situations where gdown finds the target file itself, then tries to update it with a zero-byte partial transfer in “--continue” mode. Requiring a particular suffix also helps avoid a possible source of file corruption when folders contain a lot of similarly named files.
|
@achadwick Looks really useful. Thank you for the contribution! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
--continueoption can now be used with folders.--continue(resume mode) for partially transferred folders containing lots of files..partsuffix for tempfiles, to make life nicer with Linux desktop thumbnailers.