WIP: cuml-cpu: enable conda import tests, update scikit-learn and hdbscan to match cuml

[jameslamb]

Fixes #6403

This project publishes a conda package, cuml-cpu, which does what it sounds like... allows the use of cuML on systems without a GPU.

This proposes some updates to packaging for cuml-cpu:

• enables import tests during conda builds
• updates scikit-learn and hdbscan dependencies to match cuml
• adds missing runtime dependency on packaging

Notes for Reviewers

Benefits of these Changes

This adds a bit of test coverage in CI, minimally verifying that cuml-cpu is installable an that import cuml works in an environment without a GPU.

Inspired by:

• similar changes in cuvs: cuvs-bench-cpu: avoid 'mkl' dependency on aarch64 cuvs#750
• this conversation I recently had with @betatim : rename 'sklearn' conda dependency to 'scikit-learn' cuvs#743 (comment)

It also fixes some potential user-facing bugs.

• missing dependency on packaging: WIP: cuml-cpu: enable conda import tests, update scikit-learn and hdbscan to match cuml #6400 (comment)
• cuml-cpu not installable on Python 3.12 or later because of outdated scikit-learn pin: WIP: cuml-cpu: enable conda import tests, update scikit-learn and hdbscan to match cuml #6400 (comment)

How I tested this

[jameslamb]

These changes make cuml-cpu match the versions cuml is tested against.

cuml/dependencies.yaml

Line 441 in fbcaf12

- &scikit_learn scikit-learn==1.5.*

cuml/dependencies.yaml

Line 484 in fbcaf12

- hdbscan>=0.8.39,<0.8.40

[jameslamb]

And might help with #6340

[jameslamb]

One less thing to change in a possible future where this project drops CUDA 11 support. This isn't strictly necessary for this PR but seemed small enough that it'd be ok to include here... let me know if you'd like it removed.

[jameslamb]

🎉 these import tests caught a bug!

    from cuml.internals.device_support import GPU_ENABLED
  File "device_support.pyx", line 18, in init cuml.internals.device_support
ModuleNotFoundError: No module named 'packaging'
WARNING: Tests failed for cuml-cpu-25.04.00a95-py310_250306_g2c8f9071e_95.conda - moving package to /opt/conda/conda-bld/broken

(build link)

cuml and cuml-cpu have a hard runtime dependency on packaging, from stuff like this:

cuml/python/cuml/cuml/internals/import_utils.py

Line 18 in fbcaf12

from packaging.version import Version

but don't declare that in conda packages.

The wheels do correctly declare it:

cuml/python/cuml/pyproject.toml

Line 110 in fbcaf12

"packaging",

[jameslamb]

This revealed a significant problem... the latest Python that conda-forge's scikit-learn 1.2 supports is Python 3.11: https://anaconda.org/conda-forge/scikit-learn/files?version=1.2.2

So this scikit-learn=1.2 pin could cause issues for users trying to install cuml-cpu on Python 3.12 or later.

Reproducible example:

docker run \
    --rm \
    -it rapidsai/ci-conda \
    bash

conda create \
    --yes \
    --name cuml-test \
    'cuml-cpu==25.4.*' \
    'python=3.12'

LibMambaUnsatisfiableError: Encountered problems while solving:
  - package cuml-cpu-25.04.00a80-py312_250227_g4e4f7d89e_80 requires scikit-learn 1.2.*, but none of the providers can be installed

Could not solve for environment specs
The following packages are incompatible
├─ cuml-cpu =25.4 * is installable with the potential options
│  ├─ cuml-cpu [25.04.00a80|25.04.00a81|...|25.04.00a93] would require
│  │  └─ python_abi =3.10 *_cp310, which can be installed;
│  ├─ cuml-cpu [25.04.00a80|25.04.00a81|...|25.04.00a93] would require
│  │  └─ python_abi =3.11 *_cp311, which can be installed;
│  └─ cuml-cpu [25.04.00a80|25.04.00a81|...|25.04.00a93] would require
│     └─ scikit-learn =1.2 * with the potential options
│        ├─ scikit-learn [1.2.0|1.2.1|1.2.2] would require
│        │  └─ python_abi =3.10 *_cp310, which can be installed;
│        ├─ scikit-learn [1.2.0|1.2.1|1.2.2] would require
│        │  └─ python_abi =3.11 *_cp311, which can be installed;
│        ├─ scikit-learn [1.2.0|1.2.1|1.2.2] would require
│        │  └─ python_abi =3.8 *_cp38, which can be installed;
│        └─ scikit-learn [1.2.0|1.2.1|1.2.2] would require
│           └─ python_abi =3.9 *_cp39, which can be installed;
└─ python =3.12 * is not installable because it requires
   └─ python_abi =3.12 *_cp312, which conflicts with any installable versions previously reported.

[jameslamb]

/ok to test