Uploading state snapshots to an object store

[tillrohrmann]

Ideally, we support a wide range of different object stores and filesystems. That way users can run Restate also w/o an object store. Moreover, the upload process should happen in the background.

Libraries to look at could be: https://crates.io/crates/opendal and https://crates.io/crates/object_store.

Tasks

Beta Give feedback

1. Handle existing snapshot at same
2. Configurable local snapshot file retention policy (i.e. delete after upload)

[Xuanwo]

As mentioned in #1882 (comment), please let me know if there's anything we can do from the opendal side.

We have recently polished our VISION to One Layer, All Storage with the Open Community, Solid Foundation, Fast Access, Object Storage First, and Extensible Architecture principles. We are also actively working towards our v1.0 release.