Write zarr data in rust, and exporting the result to python

[aseyboldt]

Thank you for this wonderful library!

I'm trying to figure out if I can use zarrs in a project of mine (https://github.com/pymc-devs/nutpie), but I'm not sure if what I need is currently possible:

nutpie is written in rust, and generates datasets that users interact with as a xarray.DataSet from python.

To accomplish this, I'm currently using an intermediate arrow datastructure: The rust code can create the dataset using the arrow rust code, which can then be exported through an c-ffi to python, where I convert it to xarray. This involves some copying however, and also doesn't support different storage backends, so that all datasets have to fit into memory.

Since I want to have a xarray dataset in the end anyway, I think it would be great if I could write the dataset using zarrs from rust. The problem I have is that I don't know if I can somehow pass the store to python? If I use a filesystem store, I could just write it using zarrs, close it once it is finished and open in on the python side again, but for an in-memory store that isn't an option.

zarrs-python seems to almost do what I need, but at least the documentation assumes I just want to replace the backend with with rust code globally, which doesn't help me to transfer the storage itself. (And as a library, I also don't want to modify some global behavior).

[LDeakin]

I think something like this could be supported in zarrs-python. Here is one possible way:

1. Add the capability to wrap a zarrs store with zarr.abc.store.Store
2. If a store is backed by zarrs when creating a ZarrsCodecPipeline, clone it and use it instead of creating a new one based on the store parameters

This way you would be in control of creating the store and doing what you want with it on the Python or Rust side. Would you be interested in contributing something like that?

Another alternative is enabling a zarr-python backed store to be used from Rust (the last item in ilan-gold/zarrs-python#44), but I think that would be less performant.

CC: @ilan-gold @flying-sheep

[aseyboldt]

I'm not sure I understand how that would work yet. I might be completely wrong somewhere here, but as fast as I understand things righ now: zarrs-python is a python extension module, and nutpie is a separate python extension module. Now let's assume nutpie uses zarrs to create a storage. How can zarrs-python use this storage (assuming safe rust)? It might have been built against a completely different version of zarrs.

I guess a zarr.abc.store.Store implementation could live in the zarrs crate itself behind a pyo3 feature or so? But wouldn't the CodexPipeline also have to live there then?