feat: add custom data type support (#153)

.github/workflows/ci.yml

@@ -30,6 +30,7 @@ jobs:
       - uses: Swatinem/rust-cache@v2
       - run: cargo build ${{ matrix.features }}
       - run: cargo test ${{ matrix.features }}
+      - run: cargo test ${{ matrix.features }} --examples
   build_and_test_windows:
     runs-on: windows-latest
     steps:

CHANGELOG.md

@@ -11,6 +11,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Add `array:codec::{InvalidBytesLengthError,InvalidArrayShapeError,InvalidNumberOfElementsError,SubsetOutOfBoundsError}`
 - Add `ArraySubset::inbounds_shape()` (matches the old `ArraySubset::inbounds` behaviour)
 - Add `ArrayBytesFixedDisjointView[CreateError]`
+- Add support for data type extensions with `zarrs_data_type` 0.2.0
+- Add `custom_data_type_fixed_size` and `custom_data_type_variable_size` examples
 
 ### Changed
 - **Breaking**: change `ArraySubset::inbounds` to take another subset rather than a shape
@@ -36,6 +38,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - `zarrs_plugin` is re-exported as `zarrs::plugin`
   - **Breaking**: `Plugin` is now generic over the creation arguments
   - **Breaking**: `StorageTransformerPlugin` now uses a `Plugin`
+- Add `DataTypeExtension` variant to `CodecError`
+- `ArrayCreateError::DataTypeCreateError` now uses a `PluginCreateError` internally
 
 ### Fixed
 - Fixed reserving one more element than necessary when retrieving `string` or `bytes` array elements

Cargo.toml

@@ -29,7 +29,7 @@ missing_panics_doc = "warn"
 missing_errors_doc = "warn"
 
 [workspace.dependencies.zarrs_data_type]
-version = "0.1.0"
+version = "0.2.0"
 path = "zarrs_data_type"
 
 [workspace.dependencies.zarrs_metadata]

README.md

@@ -8,17 +8,26 @@
 [![codecov](https://codecov.io/gh/LDeakin/zarrs/graph/badge.svg?token=OBKJQNAZPP)](https://codecov.io/gh/LDeakin/zarrs)
 [![DOI](https://zenodo.org/badge/695021547.svg)](https://zenodo.org/badge/latestdoi/695021547)
 
-`zarrs` is a Rust library for the [Zarr] storage format for multidimensional arrays and metadata. It supports [Zarr V3] and a V3 compatible subset of [Zarr V2].
+`zarrs` is a Rust library for the [Zarr] storage format for multidimensional arrays and metadata.
 
-A changelog can be found [here][CHANGELOG].
-Correctness issues with past versions are [detailed here][correctness_issues].
-
-Developed at the [Department of Materials Physics, Australian National University, Canberra, Australia].
+`zarrs` supports [Zarr V3] and a V3 compatible subset of [Zarr V2].
+It is fully up-to-date and conformant with the Zarr 3.0 specification with support for:
+- all *core extensions* (data types, codecs, chunk grids, chunk key encodings, storage transformers),
+- all accepted [Zarr Enhancement Proposals (ZEPs)](https://zarr.dev/zeps/) and several draft ZEPs:
+  - ZEP 0003: Variable chunking
+  - ZEP 0007: Strings
+- experimental codecs and data types intended for standardisation, and
+- user-defined custom extensions and stores.
 
 > [!TIP]
 > If you are a Python user, check out [`zarrs-python`].
 > It includes a high-performance codec pipeline for the reference [`zarr-python`] implementation.
 
+A changelog can be found [here][CHANGELOG].
+Correctness issues with past versions are [detailed here][correctness_issues].
+
+Developed at the [Department of Materials Physics, Australian National University, Canberra, Australia].
+
 ## Getting Started
 - Review the [implementation status] ([zarr version support], [array support], [storage support], and the [`zarrs` ecosystem](#zarrs-ecosystem)).
 - Read [The `zarrs` Book].
@@ -93,7 +102,7 @@ println!("{array_ndarray:4}");
 - [`zarrs`]: The core library for manipulating Zarr hierarchies.
 - [`zarrs_data_type`]: Zarr data types (re-exported as `zarrs::data_type`).
 - [`zarrs_metadata`]: Zarr metadata support (re-exported as `zarrs::metadata`).
-- [`zarrs_plugin`]: Zarr plugin support (re-exported as `zarrs::plugin`).
+- [`zarrs_plugin`]: The plugin API for `zarrs` (re-exported as `zarrs::plugin`).
 - [`zarrs_storage`]: The storage API for `zarrs` (re-exported as `zarrs::storage`).
 
 ### Stores

makefile

@@ -7,6 +7,7 @@ build:
 
 test:
 	cargo +$(TOOLCHAIN) test --all-features
+	cargo +$(TOOLCHAIN) test --all-features --examples
 
 doc: RUSTDOCFLAGS="-D warnings --cfg docsrs"
 doc:

zarrs/doc/ecosystem.md

@@ -2,7 +2,7 @@
 - [`zarrs`]: The core library for manipulating Zarr hierarchies.
 - [`zarrs_data_type`]: Zarr data types (re-exported as `zarrs::data_type`).
 - [`zarrs_metadata`]: Zarr metadata support (re-exported as `zarrs::metadata`).
-- [`zarrs_plugin`]: Zarr plugin support (re-exported as `zarrs::plugin`).
+- [`zarrs_plugin`]: The plugin API for `zarrs` (re-exported as `zarrs::plugin`).
 - [`zarrs_storage`]: The storage API for `zarrs` (re-exported as `zarrs::storage`).
 
 #### Stores
@@ -27,24 +27,24 @@
   - Transform arrays: crop, rescale, downsample, gradient magnitude, gaussian, noise filtering, etc.
   - Benchmarking tools and performance benchmarks of `zarrs`.
 
-[`zarrs`]: https://github.com/LDeakin/zarrs/tree/main/zarrs
-[`zarrs_data_type`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_data_type
-[`zarrs_metadata`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_metadata
-[`zarrs_plugin`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_plugin
-[`zarrs_storage`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_storage
-[`zarrs_filesystem`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_filesystem
-[`zarrs_http`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_http
-[`zarrs_object_store`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_object_store
-[`zarrs_opendal`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_opendal
-[`zarrs_zip`]: https://github.com/LDeakin/zarrs/tree/main/zarrs_zip
-[`zarrs_icechunk`]: https://github.com/LDeakin/zarrs_icechunk
-[`zarrs_ffi`]: https://github.com/LDeakin/zarrs_ffi
+[`zarrs`]: https://docs.rs/zarrs/latest/zarrs/
+[`zarrs_data_type`]: https://docs.rs/zarrs_data_type/latest/zarrs_data_type/
+[`zarrs_metadata`]: https://docs.rs/zarrs_metadata/latest/zarrs_metadata/
+[`zarrs_plugin`]: https://docs.rs/zarrs_plugin/latest/zarrs_plugin/
+[`zarrs_storage`]: https://docs.rs/zarrs_storage/latest/zarrs_storage/
+[`zarrs_filesystem`]: https://docs.rs/zarrs_filesystem/latest/zarrs_filesystem/
+[`zarrs_http`]: https://docs.rs/zarrs_http/latest/zarrs_http/
+[`zarrs_object_store`]: https://docs.rs/zarrs_object_store/latest/zarrs_object_store/
+[`zarrs_opendal`]: https://docs.rs/zarrs_opendal/latest/zarrs_opendal/
+[`zarrs_zip`]: https://docs.rs/zarrs_zip/latest/zarrs_zip/
+[`zarrs_icechunk`]: https://docs.rs/zarrs_icechunk/latest/zarrs_icechunk/
+[`zarrs_ffi`]: https://docs.rs/zarrs_ffi/latest/zarrs_ffi/
 [`zarrs-python`]: https://github.com/ilan-gold/zarrs-python
 [`zarr-python`]: https://github.com/zarr-developers/zarr-python
-[`zarrs_tools`]: https://github.com/LDeakin/zarrs_tools
-[`ome_zarr_metadata`]: https://github.com/LDeakin/rust_ome_zarr_metadata
-[`object_store`]: https://github.com/apache/arrow-rs/tree/main/object_store
-[`opendal`]: https://github.com/apache/OpenDAL
+[`zarrs_tools`]: https://docs.rs/zarrs_tools/latest/zarrs_tools/
+[`ome_zarr_metadata`]: https://docs.rs/ome_zarr_metadata/latest/ome_zarr_metadata/
+[`object_store`]: https://docs.rs/object_store/latest/object_store/
+[`opendal`]: https://docs.rs/opendal/latest/opendal/
 [`icechunk`]: https://github.com/earth-mover/icechunk
 
 [OME-Zarr]: https://ngff.openmicroscopy.org/latest/

zarrs/doc/status/data_types.md

@@ -4,7 +4,7 @@
 [r* (raw bits)] | [ZEP0001] | ✓ | | |
 | [bfloat16] | [zarr-specs #130] | ✓ | | |
 | [string] (experimental) | [ZEP0007 (draft)] | ✓ | | |
-| [dtype_bytes] (experimental) | [ZEP0007 (draft)] | ✓ | | |
+| [bytes](crate::data_type::DataType::Bytes) (experimental) | [ZEP0007 (draft)] | ✓ | | |
 
 <sup>† Experimental data types are recommended for evaluation only.</sup>
 
@@ -25,7 +25,6 @@
 [bfloat16]: crate::data_type::DataType::BFloat16
 [r* (raw bits)]: crate::data_type::DataType::RawBits
 [string]: crate::data_type::DataType::String
-[dtype_bytes]: crate::data_type::DataType::Bytes
 
 [ZEP0001]: https://zarr.dev/zeps/accepted/ZEP0001.html
 [zarr-specs #130]: https://github.com/zarr-developers/zarr-specs/issues/130