Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add data retention policy #188

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Add data retention policy #188

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
66d3de7
Initial commit for data retention policy discussion
asmacdo Aug 6, 2024
74fba57
Update doc/design/data-retention-policy.md
asmacdo Aug 12, 2024
cd4aee0
Apply suggestions from code review
asmacdo Aug 19, 2024
badd7ea
Apply suggestions from code review
asmacdo Aug 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 58 additions & 0 deletions doc/design/data-retention-policy.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# Data Retention Policy

Dandihub data storage on AWS EFS is expensive, and we suppose that significant portions of the data
currently stored are no longer used. Data migration is where the cost becomes extreme.
Comment on lines +3 to +4
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since S3 buckets can now mount to EC2 instances (reference: August 2023 blog post) and S3 costs are ~10X cheaper than EFS, as part of this data retention work perhaps we should also look into what it would take to move to S3 storage (and discuss any features that would not be available with this migration)?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Filed #198


## Persistent Data locations
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
## Persistent Data locations
## Persistent Data Locations


Each user has access to 2 locations: `/home/{user}` and `/shared/`.

Within jupyterhub `/home/{user}` the user always sees `/home/jovyan`, but is stored in EFS as their GitHub
username.
asmacdo marked this conversation as resolved.
Show resolved Hide resolved
Comment on lines +8 to +11
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We were previously considering providing a /scratch directory for each user that is automatically cleaned up after 30 days. In addition to the policy for the /home/jovyan directory, do we also want to implement a /scratch directory with a 30 day clean up policy?


## Known cache file cleanup
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
## Known cache file cleanup
## Known Cache File Cleanup


We should be able to safely remove the following:
kabilar marked this conversation as resolved.
Show resolved Hide resolved
- `/home/{user}/.cache`
- `nwb_cache`
- Yarn Cache
- `__pycache__`
- pip cache
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case user is still active -- I think it would be useful to report to the long running users, after reaching some threshold on any of those folders (e.g. 50MB) asking to clean them up.

Copy link
Member

@kabilar kabilar Sep 17, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @asmacdo, should we add a separate point here about monitoring and reporting the quotas of cache directories for active users?



## Determining Last Access

EFS does not store metadata for the last access of the data. (Though they must track somehow to move to `Infrequent Access`)

Alternatives:
- use the [jupyterhub REST API](https://jupyterhub.readthedocs.io/en/stable/reference/rest-api.html#operation/get-users) check when user last used/logged in to hub.
- dandiarchive login information

## Automated Data Audit

Copy link
Member

@kabilar kabilar Sep 17, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
At an interval of 7 days:
- Calculate home directory disk usage

At some interval (30 days with no login?):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
At some interval (30 days with no login?):
At an interval of 30 days with no login to JupyterHub:

- find files larger than 100 (?) GB and mtime > 10 (?) days -- get total size and count
- find files larger than 1 (?) GB and mtime > 30 (?) days -- get total size and count
asmacdo marked this conversation as resolved.
Show resolved Hide resolved
- find _pycache_ and nwb-cache folders and pip cache and mtime > 30? days -- total sizes and list of them
Comment on lines +34 to +36
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- find files larger than 100 (?) GB and mtime > 10 (?) days -- get total size and count
- find files larger than 1 (?) GB and mtime > 30 (?) days -- get total size and count
- find _pycache_ and nwb-cache folders and pip cache and mtime > 30? days -- total sizes and list of them
- find files larger than 100 GB and mtime > 10 days -- get total size and count
- find files larger than 1 GB and mtime > 30 days -- get total size and count
- find _pycache_ and nwb-cache folders and pip cache and mtime > 30 days -- total sizes and list of them


Notify user if:
- any of the above listed thresholds were reached
- total du exceeds some threshold (e.g. 100G)
asmacdo marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- total du exceeds some threshold (e.g. 100G)
- total home directory disk usage exceeds 1 TB

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggested a quota of 1 TB for home directories as many datasets are getting to be quite large. This would provide temporary, high-capacity storage, but hopefully users won't get anywhere near this threshold. This would cost $300/user/month for standard EFS, and $23/user/month if we move to Standard S3.

If we implement a scratch directory, then perhaps the home directory can have a much smaller quota.

- total outdated caches size exceeds some threshold (e.g. 1G)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- total outdated caches size exceeds some threshold (e.g. 1G)
- total outdated caches size exceeds 1 GB

- prior notification was sent more than a week ago

Notification information:
- large file list
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- large file list
- summarized audit data (total size and count for each of the above thresholds)
- large file list

- summarized data retention policy
- Notice number
- request to cleanup
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

meanwhile it might be worth creating a simple data record schema to store those records as well so they could be reused by the tools to assemble higher level stats etc.


### Non-response cleanup

If a user has not logged in for 60 days (30 days initial + 30 days following audit), send a warning:
`In 10 days the following files will be cleaned up`

If the user has not logged in for 70 days (30 initial + 30 after audit + 10 warning):
`The following files were removed`

Reset timer.