Update dependency datasets to v3.3.2

[renovate]

This PR contains the following updates:

Package	Change	Age	Adoption	Passing	Confidence
datasets	3.1.0 -> 3.3.2

---

Release Notes

huggingface/datasets (datasets)

v3.3.2

Compare Source

Bug fixes

• Attempt to fix multiprocessing hang by closing and joining the pool before termination by @​dakinggg in https://github.com/huggingface/datasets/pull/7411
• Gracefully cancel async tasks by @​lhoestq in https://github.com/huggingface/datasets/pull/7414

Other general improvements

• Update use_with_pandas.mdx: to_pandas() correction in last section by @​ibarrien in https://github.com/huggingface/datasets/pull/7407
• Fix a typo in arrow_dataset.py by @​jingedawang in https://github.com/huggingface/datasets/pull/7402

New Contributors

• @​dakinggg made their first contribution in https://github.com/huggingface/datasets/pull/7411
• @​ibarrien made their first contribution in https://github.com/huggingface/datasets/pull/7407
• @​jingedawang made their first contribution in https://github.com/huggingface/datasets/pull/7402

Full Changelog: huggingface/datasets@3.3.1...3.3.2

v3.3.1

Compare Source

Bug fixes

• Fix filter speed regression by @​lhoestq in https://github.com/huggingface/datasets/pull/7408

Full Changelog: huggingface/datasets@3.3.0...3.3.1

v3.3.0

Compare Source

Dataset Features

• Support async functions in map() by @​lhoestq in https://github.com/huggingface/datasets/pull/7384

  • Especially useful to download content like images or call inference APIs

  prompt = "Answer the following question: {question}. You should think step by step."
  async def ask_llm(example):
      return await query_model(prompt.format(question=example["question"]))
  ds = ds.map(ask_llm)

• Add repeat method to datasets by @​alex-hh in https://github.com/huggingface/datasets/pull/7198

  ds = ds.repeat(10)

• Support faster processing using pandas or polars functions in IterableDataset.map() by @​lhoestq in https://github.com/huggingface/datasets/pull/7370

  • Add support for "pandas" and "polars" formats in IterableDatasets
  • This enables optimized data processing using pandas or polars functions with zero-copy, e.g.

  ds = load_dataset("ServiceNow-AI/R1-Distill-SFT", "v0", split="train", streaming=True)
  ds = ds.with_format("polars")
  expr = pl.col("solution").str.extract("boxed\\{(.*)\\}").alias("value_solution")
  ds = ds.map(lambda df: df.with_columns(expr), batched=True)

• Apply formatting after iter_arrow to speed up format -> map, filter for iterable datasets by @​alex-hh in https://github.com/huggingface/datasets/pull/7207

  • IterableDatasets with "numpy" format are now much faster

What's Changed

• don't import soundfile in tests by @​lhoestq in https://github.com/huggingface/datasets/pull/7340
• minor video docs on how to install by @​lhoestq in https://github.com/huggingface/datasets/pull/7341
• Fix typo in arrow_dataset by @​AndreaFrancis in https://github.com/huggingface/datasets/pull/7328
• remove filecheck to enable symlinks by @​fschlatt in https://github.com/huggingface/datasets/pull/7133
• Webdataset special columns in last position by @​lhoestq in https://github.com/huggingface/datasets/pull/7349
• Bump hfh to 0.24 to fix ci by @​lhoestq in https://github.com/huggingface/datasets/pull/7350
• fsspec 2024.12.0 by @​lhoestq in https://github.com/huggingface/datasets/pull/7352
• changes to MappedExamplesIterable to resolve #​7345 by @​vttrifonov in https://github.com/huggingface/datasets/pull/7353
• Catch OSError for arrow by @​lhoestq in https://github.com/huggingface/datasets/pull/7348
• Remove .h5 from imagefolder extensions by @​lhoestq in https://github.com/huggingface/datasets/pull/7374
• Add Pandas, PyArrow and Polars docs by @​lhoestq in https://github.com/huggingface/datasets/pull/7382
• Optimized sequence encoding for scalars by @​lukasgd in https://github.com/huggingface/datasets/pull/7393
• Update docs by @​lhoestq in https://github.com/huggingface/datasets/pull/7395
• Update README.md by @​lhoestq in https://github.com/huggingface/datasets/pull/7396
• Release: 3.3.0 by @​lhoestq in https://github.com/huggingface/datasets/pull/7398

New Contributors

• @​AndreaFrancis made their first contribution in https://github.com/huggingface/datasets/pull/7328
• @​vttrifonov made their first contribution in https://github.com/huggingface/datasets/pull/7353
• @​lukasgd made their first contribution in https://github.com/huggingface/datasets/pull/7393

Full Changelog: huggingface/datasets@3.2.0...3.3.0

v3.2.0

Compare Source

Dataset Features

• Faster parquet streaming + filters with predicate pushdown by @​lhoestq in https://github.com/huggingface/datasets/pull/7309
  • Up to +100% streaming speed
  • Fast filtering via predicate pushdown (skip files/row groups based on predicate instead of downloading the full data), e.g.

    from datasets import load_dataset
    filters = [('date', '>=', '2023')]
    ds = load_dataset("HuggingFaceFW/fineweb-2", "fra_Latn", streaming=True, filters=filters)

Other improvements and bug fixes

• fix conda release worlflow by @​lhoestq in https://github.com/huggingface/datasets/pull/7272
• Add link to video dataset by @​NielsRogge in https://github.com/huggingface/datasets/pull/7277
• Raise error for incorrect JSON serialization by @​varadhbhatnagar in https://github.com/huggingface/datasets/pull/7273
• support for custom feature encoding/decoding by @​alex-hh in https://github.com/huggingface/datasets/pull/7284
• update load_dataset doctring by @​lhoestq in https://github.com/huggingface/datasets/pull/7301
• Let server decide default repo visibility by @​Wauplin in https://github.com/huggingface/datasets/pull/7302
• fix: update elasticsearch version by @​ruidazeng in https://github.com/huggingface/datasets/pull/7300
• Fix typing in iterable_dataset.py by @​lhoestq in https://github.com/huggingface/datasets/pull/7304
• Updated inconsistent output in documentation examples for ClassLabel by @​sergiopaniego in https://github.com/huggingface/datasets/pull/7293
• More docs to from_dict to mention that the result lives in RAM by @​lhoestq in https://github.com/huggingface/datasets/pull/7316
• Release: 3.2.0 by @​lhoestq in https://github.com/huggingface/datasets/pull/7317

New Contributors

• @​ruidazeng made their first contribution in https://github.com/huggingface/datasets/pull/7300
• @​sergiopaniego made their first contribution in https://github.com/huggingface/datasets/pull/7293

Full Changelog: huggingface/datasets@3.1.0...3.2.0

---

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.