[BUG] cuml-cpu fails to import without a GPU present

[jameslamb]

Describe the bug

Working on #6400, I encountered multiple errors when trying to import cuml installed via the cuml-cpu conda package in an environment without a GPU.

Documenting those cases here.

Steps/Code to reproduce bug

Ran the following on an M2 (arm64) macbook pro:

docker run \
    --rm \
    -it rapidsai/ci-conda:latest \
    bash

conda create \
    --name cuml-test \
    'cuml-cpu==25.4.*' \
    'python=3.11'

source activate cuml-test

Environment creation succeeded.

details (click me)

Output of conda info

     active environment : cuml-test                                                                                                                                                                     
    active env location : /opt/conda/envs/cuml-test                                                                                                                                                     
            shell level : 2                                                                                                                                                                             
       user config file : /root/.condarc                                                                                                                                                                
 populated config files : /opt/conda/.condarc                                                                                                                                                           
          conda version : 24.7.1                                                                                                                                                                        
    conda-build version : 24.5.1                                                                                                                                                                        
         python version : 3.12.5.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=aarch64
                          __conda=24.7.1=0
                          __glibc=2.35=0
                          __linux=6.5.0=0
                          __unix=0=0
       base environment : /opt/conda  (writable)
      conda av data dir : /opt/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/rapidsai/linux-aarch64
                          https://conda.anaconda.org/rapidsai/noarch
                          https://conda.anaconda.org/rapidsai-nightly/linux-aarch64
                          https://conda.anaconda.org/rapidsai-nightly/noarch
                          https://conda.anaconda.org/dask/label/dev/linux-aarch64
                          https://conda.anaconda.org/dask/label/dev/noarch
                          https://conda.anaconda.org/pytorch/linux-aarch64
                          https://conda.anaconda.org/pytorch/noarch
                          https://conda.anaconda.org/conda-forge/linux-aarch64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/nvidia/linux-aarch64
                          https://conda.anaconda.org/nvidia/noarch
          package cache : /opt/conda/pkgs
                          /root/.conda/pkgs
       envs directories : /opt/conda/envs
                          /root/.conda/envs
               platform : linux-aarch64
             user-agent : conda/24.7.1 requests/2.32.3 CPython/3.12.5 Linux/6.5.0-44-generic ubuntu/22.04.5 glibc/2.35 solver/libmamba conda-libmamba-solver/24.7.0 libmambapy/1.5.9
                UID:GID : 0:0
             netrc file : None
           offline mode : False

Output of conda env export --name cuml-test

name: cuml-test
channels:
  - rapidsai
  - rapidsai-nightly
  - dask/label/dev
  - pytorch
  - conda-forge
  - nvidia
dependencies:
  - _openmp_mutex=4.5=2_gnu
  - bzip2=1.0.8=h68df207_7
  - ca-certificates=2025.1.31=hcefe29a_0
  - colorama=0.4.6=pyhd8ed1ab_1
  - cuml-cpu=25.04.00a95=py311_250306_gee276d5f2_95
  - hdbscan=0.8.38.post2=py311hec9beba_0
  - icu=75.1=hf9b3779_0
  - joblib=1.4.2=pyhd8ed1ab_1
  - ld_impl_linux-aarch64=2.43=h80caac9_4
  - libblas=3.9.0=31_h1a9f1db_openblas
  - libcblas=3.9.0=31_hab92f65_openblas
  - libexpat=2.6.4=h5ad3122_0
  - libffi=3.4.6=he21f813_0
  - libgcc=14.2.0=he277a41_2
  - libgcc-ng=14.2.0=he9431aa_2
  - libgfortran=14.2.0=he9431aa_2
  - libgfortran5=14.2.0=hb6113d0_2
  - libgomp=14.2.0=he277a41_2
  - libhwloc=2.11.2=default_h2c612a5_1001
  - libiconv=1.18=hc99b53d_1
  - liblapack=3.9.0=31_h411afd4_openblas
  - libllvm15=15.0.7=hcc6320c_5
  - liblzma=5.6.4=h86ecc28_0
  - libnsl=2.0.1=h31becfc_0
  - libopenblas=0.3.29=pthreads_h9d3fd7e_0
  - libsqlite=3.49.1=h5eb1b54_1
  - libstdcxx=14.2.0=h3f4de04_2
  - libstdcxx-ng=14.2.0=hf1166c9_2
  - libuuid=2.38.1=hb4cce97_0
  - libxcrypt=4.4.36=h31becfc_1
  - libxml2=2.13.6=h2e0c361_0
  - libzlib=1.3.1=h86ecc28_2
  - llvmlite=0.44.0=py311hbc3773b_0
  - ncurses=6.5=ha32ae93_3
  - numba=0.61.0=py311h26ae67a_1
  - numpy=1.26.4=py311h69ead2a_0
  - nvtx=0.2.11=py311h5487e9b_0
  - openssl=3.4.1=hd08dc88_0
  - pandas=2.2.3=py311h848c333_2
  - pip=25.0.1=pyh8b19718_0
  - pynndescent=0.5.13=pyhd8ed1ab_1
  - python=3.11.11=h1683364_2_cpython
  - python-dateutil=2.9.0.post0=pyhff2d567_1
  - python-tzdata=2025.1=pyhd8ed1ab_0
  - python_abi=3.11=5_cp311
  - pytz=2024.1=pyhd8ed1ab_0
  - readline=8.2=h8382b9d_2
  - scikit-learn=1.2.2=py311hb93614b_2
  - scipy=1.15.2=py311h2973cce_0
  - setuptools=75.8.2=pyhff2d567_0
  - six=1.17.0=pyhd8ed1ab_0
  - tbb=2022.0.0=h243be18_0
  - threadpoolctl=3.5.0=pyhc1e730c_0
  - tk=8.6.13=h194ca79_0
  - tqdm=4.67.1=pyhd8ed1ab_1
  - tzdata=2025a=h78e105d_0
  - umap-learn=0.5.6=py311hec3470c_1
  - wheel=0.45.1=pyhd8ed1ab_1
prefix: /opt/conda/envs/cuml-test

Tried to import like this:

python -c "import cuml"

That failed like this, because the package uses packaging unconditionally.

ModuleNotFoundError: No module named 'packaging'

full traceback (click me)

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/__init__.py", line 27, in <module>
    from cuml.internals.base import Base, UniversalBase
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/internals/__init__.py", line 17, in <module>
    from cuml.internals.available_devices import is_cuda_available
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/internals/available_devices.py", line 16, in <module>
    from cuml.internals.device_support import GPU_ENABLED
  File "device_support.pyx", line 18, in init cuml.internals.device_support
ModuleNotFoundError: No module named 'packaging'

So tried installing that and importing again.

conda install --yes packaging
python -c "import cuml"

That fails again, this time because of an import of cupy.

...
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/internals/safe_imports.py", line 86, in __getattr__
    raise UnavailableError(cls._msg)
cuml.internals.safe_imports.UnavailableError: cupy is not installed in non GPU-enabled installations

full traceback (click me)

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/__init__.py", line 116, in <module>
    from cuml.linear_model.linear_regression import LinearRegression
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/linear_model/__init__.py", line 21, in <module>
    from cuml.linear_model.logistic_regression import LogisticRegression
  File "logistic_regression.pyx", line 27, in init cuml.linear_model.logistic_regression
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/preprocessing/__init__.py", line 16, in <module>
    from cuml.model_selection import train_test_split
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/model_selection/__init__.py", line 17, in <module>
    from cuml.model_selection._split import train_test_split
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/model_selection/_split.py", line 41, in <module>
    indices: cp.ndarray,
             ^^^^^^^^^^
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/internals/safe_imports.py", line 86, in __getattr__
    raise UnavailableError(cls._msg)
cuml.internals.safe_imports.UnavailableError: cupy is not installed in non GPU-enabled installations

So tried installing that and importing again.

conda install --yes cupy
python -c "import cuml"

That fails again, this time because of an import of cudf

  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/internals/safe_imports.py", line 86, in __getattr__
    raise UnavailableError(cls._msg)
cuml.internals.safe_imports.UnavailableError: cudf is not installed in non GPU-enabled installations

full traceback (click me)

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/__init__.py", line 116, in <module>
    from cuml.linear_model.linear_regression import LinearRegression
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/linear_model/__init__.py", line 21, in <module>
    from cuml.linear_model.logistic_regression import LogisticRegression
  File "logistic_regression.pyx", line 27, in init cuml.linear_model.logistic_regression
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/preprocessing/__init__.py", line 17, in <module>
    from cuml.preprocessing.LabelEncoder import LabelEncoder
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/preprocessing/LabelEncoder.py", line 56, in <module>
    class LabelEncoder(Base):
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/preprocessing/LabelEncoder.py", line 212, in LabelEncoder
    def transform(self, y) -> cudf.Series:
                              ^^^^^^^^^^^
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cuml/internals/safe_imports.py", line 86, in __getattr__
    raise UnavailableError(cls._msg)
cuml.internals.safe_imports.UnavailableError: cudf is not installed in non GPU-enabled installations

Tried installing that and importing again.

conda install --yes 'cudf==25.4.*'
python -c "import cuml"

That failed like this:

numba.cuda.cudadrv.error.CudaSupportError: Error at driver init: 
...
CUDA driver library cannot be found.
...
/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cudf/utils/gpu_utils.py:62: UserWarning: Failed to dlopen libcuda.so.1

full traceback (click me)

/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cudf/utils/_ptxcompiler.py:64: UserWarning: Error getting driver and runtime versions:

stdout:

stderr:

Traceback (most recent call last):
  File "<string>", line 4, in <module>
  File "/opt/conda/envs/cuml-test/lib/python3.11/site-packages/numba_cuda/numba/cuda/cudadrv/driver.py", line 295, in __getattr__
    raise CudaSupportError("Error at driver init: \n%s:" %
numba.cuda.cudadrv.error.CudaSupportError: Error at driver init: 

CUDA driver library cannot be found.
If you are sure that a CUDA driver is installed,
try setting environment variable NUMBA_CUDA_DRIVER
with the file path of the CUDA driver shared library.
:
Not patching Numba
  warnings.warn(msg, UserWarning)
/opt/conda/envs/cuml-test/lib/python3.11/site-packages/cudf/utils/gpu_utils.py:62: UserWarning: Failed to dlopen libcuda.so.1
  warnings.warn(str(e)

Expected behavior

Expected to be able to use cuml-cpu in an environment

Environment details (please complete the following information):

See "Steps/Code to reproduce bug" section.

Additional context

This report might be related to these others:

• [BUG] Unable to run Nvidia morpheus-experimental example using cuml-cpu #6349
• [BUG] umap_utils fails to import in cuml-cpu #6341
• [BUG] cuml-cpu conda install broken #6340

From their contents, I'm not sure.