Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Evaluate database storage parameters #954

Open
KlaasH opened this issue Mar 8, 2024 · 0 comments
Open

Evaluate database storage parameters #954

KlaasH opened this issue Mar 8, 2024 · 0 comments

Comments

@KlaasH
Copy link
Contributor

KlaasH commented Mar 8, 2024

On 3/8/24 the database ran out of space and became unavailable. I increased the allocation by hand, in the console, from 120GB to 140GB and enabled storage auto-scaling. That solved the immediate problem, but as follow-up, we should:

  • Check that the storage is being used how we expect. I.e. that the storage needs have grown due to the space needed by new analysis runs, not because there's cruft or temporary data being left around.
  • Decide what parameters we want, specifically
    • Whether to keep auto-scaling enabled
    • What max limit to set, if any
    • Whether to switch from gp2 to gp3 storage
  • Change parameters in the Terraform/tfvars to match those decisions

Note: if we decide to turn off storage auto-scaling, we should adjust the threshold on the CloudWatch alarm to make it go off at a higher value. If we decide to stick with auto-scaling, an alarm isn't as critical, though it would still make sense to have one that would go off before we hit the max limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant