replace piz with rc-zip

[luizirber]

rc-zip can work on top of regular files (not memmap-ed files like piz).

It is also much easier to interact with.

Pending: more perf tests, but mem-wise I ran the command from #1909 (comment) and was seeing ~110MB max memory consumption.

---

Sadly, I think this will be draft for quite some time, it needs Rust 1.74 to compile =/
(it also broke wasm, need to do some splitting in src/core/src/storage.rs to be able to compile a subset...)

[codecov]

Codecov Report

Attention: Patch coverage is 68.42105% with 30 lines in your changes missing coverage. Please review.

Project coverage is 87.94%. Comparing base (d57039f) to head (a7561d9).

Files with missing lines	Patch %	Lines
src/core/src/storage/zip.rs	67.41%	29 Missing ⚠️
src/core/src/errors.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           latest    #3012      +/-   ##
==========================================
- Coverage   87.99%   87.94%   -0.05%     
==========================================
  Files         136      137       +1     
  Lines       22230    22214      -16     
  Branches     2225     2225              
==========================================
- Hits        19561    19536      -25     
- Misses       2356     2365       +9     
  Partials      313      313              

Flag	Coverage Δ
hypothesis-py	25.43% <ø> (ø)
python	92.32% <ø> (ø)
rust	81.82% <68.42%> (-0.13%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[bluegenes]

Sadly, I think this will be draft for quite some time, it needs Rust 1.74 to compile =/

What do we need in order to switch to 1.74? Is it just time, other packages, etc?

[luizirber]

What do we need in order to switch to 1.74? Is it just time, other packages, etc?

We don't really have any rules, but I usually check

• What is supported in debian stable? https://packages.debian.org/search?keywords=rustc
• What is supported in ubuntu? https://packages.ubuntu.com/search?keywords=rust-all&searchon=names&exact=1&suite=all&section=all
• https://repology.org/project/rust/information used to be useful, but now it is a bit messy to read.

1.74 was released Nov 16th, which is fairly recent for distro standards/release cycles. 1.74 is also a pretty big release due to projections in opaque return types, which is exactly the feature that rc-zip uses and makes async programming much easier. Not that we do async, but many, many crates are going to MSRV to 1.74 because of that...

Maybe a good rule of thumb is checking what rust is available on the places we do packaging:

• pypi - we can choose
• conda-forge - 1.76
• pyodide - nightly-2024-01-29?!?!?!?
• spack - 1.75, about to add 1.76
• nixpkgs (1.73 on latest release, 1.76 on rolling)

But these are fairly fast moving places =]

More links:

• Fairly long discussion on MSRV and linux distros: https://internals.rust-lang.org/t/rust-msrv-policy-and-linux-distros/17074
• I don't expect anyone running sourmash on embedded systems, but Ferrocene is 1.68 at the moment

[ctb]

(asking here, because it's relevant to some memory questions) - how are we feeling about Rust 1.74 these days?

[luizirber]

(asking here, because it's relevant to some memory questions) - how are we feeling about Rust 1.74 these days?

I've been using https://lib.rs/stats#rustc-usage to check MSRV bumps, 1.74 would put us in 96-97% of all crates.io requests. 1.74 was released November 16, 2023, so I think it is OK to bump MSRV.