Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove archive after it is extracted to save disk space #1351

Merged
merged 5 commits into from
Sep 29, 2024
Merged

Conversation

PGijsbers
Copy link
Collaborator

@PGijsbers PGijsbers commented Sep 13, 2024

Closes #1348

I don't really know that we need to make this configurable at this point. Let's not add more options for now, and see if we get requests.

@PGijsbers
Copy link
Collaborator Author

@prabhant This is what you are looking for, right?

@codecov-commenter
Copy link

codecov-commenter commented Sep 13, 2024

Codecov Report

Attention: Patch coverage is 16.66667% with 10 lines in your changes missing coverage. Please review.

Project coverage is 84.20%. Comparing base (7764ddb) to head (731b3e1).
Report is 1 commits behind head on develop.

Files with missing lines Patch % Lines
openml/_api_calls.py 16.66% 10 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1351      +/-   ##
===========================================
- Coverage    84.28%   84.20%   -0.09%     
===========================================
  Files           38       38              
  Lines         5288     5298      +10     
===========================================
+ Hits          4457     4461       +4     
- Misses         831      837       +6     
Flag Coverage Δ
84.20% <16.66%> (-0.09%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@prabhant
Copy link
Contributor

Looks good to me

@PGijsbers PGijsbers marked this pull request as ready for review September 27, 2024 12:56
@PGijsbers
Copy link
Collaborator Author

PGijsbers commented Sep 27, 2024

It just sprang to my mind that removing the file does conflict with the caching mechanism, so it will likely always download the archive now... What would be a good way to resolve that? Leaving some kind of empty marker with the same name could be a quick hack. @LennartPurucker @eddiebergman

@eddiebergman
Copy link
Collaborator

eddiebergman commented Sep 27, 2024

Interesting, I came across this exact problem recently where I don't know the contents of tarfiles directly and needed to check if their content were already present. Problem being I can't map the contents of the archive with the archive itself, and whether a new download should be triggered

I didn't implement a solution other than force: bool but your idea about a marker sounds promising. It would assume that the any new archive that might be downloaded are uniuly named, such that you can one-to-one map them to a marker file.

I don't have a good solution but please keep me posted if you come up with one!

Copy link
Contributor

@LennartPurucker LennartPurucker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, I agree. A marker file for the file being already extracted would be needed and an option to still force the download (force bool or cleaning up the cache).

Ideally, we would have a uuid for the content of the zip file on the server that, if it changes, would prompt us to re-download. Then, we would just name the marker file using this uuid and check for a match before downloading and extracting again.

@PGijsbers
Copy link
Collaborator Author

In this case we can apparently use the metadata which contains a hash (at least, as far as I can tell, that's what it is). For a force refresh, we already support that at a get_dataset level, so I didn't think we need to do anything special here.

Copy link
Contributor

@LennartPurucker LennartPurucker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice, LGTM, thanks!

@PGijsbers PGijsbers merged commit a3e57bb into develop Sep 29, 2024
7 of 14 checks passed
@PGijsbers PGijsbers deleted the add/cleanup branch September 29, 2024 12:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add clean up function for downloaded image archive
5 participants