Release 2.18.0 · huggingface/datasets

Dataset features

Make JSON builder support an array of strings by @albertvillanova in #6696
Base parquet batch_size on parquet row group size by @lhoestq in #6701
- Faster cold start for streaming
Change default compression argument for JsonDatasetWriter by @Rexhaif in #6659
Automatic Conversion for uint16/uint32 to Compatible PyTorch Dtypes by @mohalisad in #6660
fsspec: support fsspec>=2023.12.0 glob changes by @pmrowla in #6687
- Support latest fsspec up to 2024.2.0

Fix for Incorrect ex_iterable used with multi num_worker by @kq-chen in #6582
- Previously using PyTorch DDP and num_workers could lead to incorrect shards assignments to workers and cause errors
Fix imagefolder dataset url by @mariosasko in #6683
Improve error message for gated datasets on load by @lewtun in #6684
Updated Quickstart Notebook link by @Codeblockz in #6685
Update the print message for chunked_dataset in process.mdx by @gzbfgjf2 in #6693
Faster xlistdir by @mariosasko in #6698
Update GitHub Actions to Node 20 by @albertvillanova in #6682
Update release instructions by @albertvillanova in #6681
Pass through information about location of cache directory. by @stridge-cruxml in #6677
Allow SplitDict setitem to replace existing SplitInfo by @lhoestq in #6665
Update ruff by @lhoestq in #6706
Silence ruff deprecation messages by @mariosasko in #6707
fix: show correct package name to install biopython by @BioGeek in #6662
Fix data_files when passing data_dir by @lhoestq in #6705
Release: 2.18.0 by @lhoestq in #6708

Full Changelog: 2.17.1...2.18.0