Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a reliable "write to cloud storage" archive service implementation #589

Open
Tracked by #357
jsync-swirlds opened this issue Feb 5, 2025 · 0 comments
Open
Tracked by #357

Comments

@jsync-swirlds
Copy link
Member

jsync-swirlds commented Feb 5, 2025

Create a new archive class service, which may deploy in addition to other archive service(s).
This archive service will, based on separate configuration, take archived packages of block files (ideally quite large files) and reliably migrate those files to cloud storage.

Suggested features

  • Support for both GCP cloud storage and Amazon S3
  • Direct API calls, without use of cloud provider SDKs
    • Handle and retry for all non-fatal errors.
  • Verify-before-remove
    • calculate and verify a file hash (e.g. MD5 or SHA2) and verify against the hash calculated by the cloud storage provider.
    • Only remove the local file if the hash values match.
    • Retry if the hash values do not match.
  • Upload on background Virtual Threads.
@jsync-swirlds jsync-swirlds changed the title Add a reliable "write to remote archive" archive service implementation Add a reliable "write to cloud storage" archive service implementation Feb 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant