Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor codec metadata and fill values #157

Merged
merged 10 commits into from
Mar 9, 2025
Merged

Refactor codec metadata and fill values #157

merged 10 commits into from
Mar 9, 2025

Conversation

LDeakin
Copy link
Owner

@LDeakin LDeakin commented Mar 8, 2025

This is a massive refactor, but should not be particularly impactful to users besides codec metadata moving to a new module.

zarrs_metadata

  • Move all codecs into a new codec module
    • zarrs_metadata no longer distinguishes Zarr V2/V3 codecs within the v2/v3 modules
    • Zarr V2/V3 codec aliasing is handled in zarrs
  • Add zfpy codec metadata (unmerging zfpy/zfp)
    • Remove write_header from zfp codec configuration (since zfpy is now separate)
  • Refactor FillValueMetadataV3 to support arbitrary fill value metadata for ZEP0009

zarrs

  • Change the default vlen codecs to vlen-utf8 / vlen-bytes instead of experimental vlen
  • Refactor codec name handling and CodecTraits in alignment with ZEP0009 and the zarr-extensions repository
    • All "experimental" codecs now use the zarrs. prefix (or numcodecs. if fully compatible) instead of a URL
    • Add support for aliased codec names
  • Improve the codec docs

Closes #116
Closes #125

@LDeakin LDeakin added documentation Improvements or additions to documentation enhancement New feature or request breaking-changes The PR has API changes compatibility Related to ecosystem compatibility labels Mar 8, 2025
@LDeakin LDeakin added this to the zarrs 0.20 milestone Mar 8, 2025
Copy link

codecov bot commented Mar 8, 2025

Codecov Report

Attention: Patch coverage is 86.24161% with 205 lines in your changes missing coverage. Please review.

Project coverage is 81.91%. Comparing base (74bee58) to head (27eb79e).
Report is 11 commits behind head on main.

Files with missing lines Patch % Lines
.../array_to_bytes/sharding/sharding_codec_builder.rs 30.61% 34 Missing ⚠️
zarrs/src/array/codec/options.rs 32.25% 21 Missing ⚠️
zarrs/src/config.rs 34.78% 15 Missing ⚠️
...ay/codec/array_to_array/bitround/bitround_codec.rs 33.33% 14 Missing ⚠️
...ay/codec/bytes_to_bytes/gdeflate/gdeflate_codec.rs 33.33% 14 Missing ⚠️
...arrs/src/array/codec/array_to_bytes/codec_chain.rs 89.36% 10 Missing ⚠️
...rs/src/array/codec/array_to_bytes/zfp/zfp_codec.rs 71.42% 10 Missing ⚠️
zarrs_metadata/src/v3/array/fill_value.rs 94.65% 10 Missing ⚠️
zarrs_metadata/src/codec_map.rs 62.50% 9 Missing ⚠️
zarrs_metadata/src/v2_to_v3.rs 85.96% 8 Missing ⚠️
... and 20 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #157      +/-   ##
==========================================
- Coverage   81.98%   81.91%   -0.07%     
==========================================
  Files         177      179       +2     
  Lines       25048    25522     +474     
==========================================
+ Hits        20535    20907     +372     
- Misses       4513     4615     +102     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@LDeakin LDeakin force-pushed the codec_aliasing branch 2 times, most recently from 3ed1207 to ac0537a Compare March 9, 2025 00:02
- move CodecMetadataOptions
- make `array::codec::{[metadata_]options}` private
@LDeakin LDeakin marked this pull request as ready for review March 9, 2025 06:09
@LDeakin LDeakin merged commit 6d3cc2d into main Mar 9, 2025
22 checks passed
@LDeakin LDeakin deleted the codec_aliasing branch March 9, 2025 06:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking-changes The PR has API changes compatibility Related to ecosystem compatibility documentation Improvements or additions to documentation enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Mark relevant Configuration enums as #[non_exhaustive] Specs for experimental codecs
1 participant