[Installation]: no version of pip install vllm works - Failed to initialize NumPy: No Module named 'numpy'

[cephdon]

Your current environment

Traceback (most recent call last):
  File "/mnt/MSAI/home/cephdon/sources/vllm/collect_env.py", line 15, in <module>
    from vllm.envs import environment_variables
  File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/__init__.py", line 3, in <module>
    from vllm.engine.arg_utils import AsyncEngineArgs, EngineArgs
  File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/engine/arg_utils.py", line 11, in <module>
    from vllm.config import (CacheConfig, CompilationConfig, ConfigFormat,
  File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/config.py", line 21, in <module>
    from vllm.model_executor.layers.quantization import (QUANTIZATION_METHODS,
  File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/model_executor/__init__.py", line 1, in <module>
    from vllm.model_executor.parameter import (BasevLLMParameter,
  File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/model_executor/parameter.py", line 7, in <module>
    from vllm.distributed import get_tensor_model_parallel_rank
  File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/distributed/__init__.py", line 1, in <module>
    from .communication_op import *
  File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/distributed/communication_op.py", line 6, in <module>
    from .parallel_state import get_tp_group
  File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/distributed/parallel_state.py", line 38, in <module>
    import vllm.distributed.kv_transfer.kv_transfer_agent as kv_transfer
  File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/distributed/kv_transfer/kv_transfer_agent.py", line 15, in <module>
    from vllm.distributed.kv_transfer.kv_connector.factory import (
  File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/distributed/kv_transfer/kv_connector/factory.py", line 3, in <module>
    from .base import KVConnectorBase
  File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/distributed/kv_transfer/kv_connector/base.py", line 14, in <module>
    from vllm.sequence import IntermediateTensors
  File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/sequence.py", line 16, in <module>
    from vllm.inputs import SingletonInputs, SingletonInputsAdapter
  File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/inputs/__init__.py", line 7, in <module>
    from .registry import (DummyData, InputContext, InputProcessingContext,
  File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/inputs/registry.py", line 13, in <module>
    from vllm.transformers_utils.tokenizer import AnyTokenizer
  File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/transformers_utils/tokenizer.py", line 16, in <module>
    from vllm.utils import make_async
  File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/utils.py", line 44, in <module>
    from vllm.platforms import current_platform
  File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/platforms/__init__.py", line 100, in <module>
    from .cuda import CudaPlatform
  File "/mnt/MSAI/home/cephdon/sources/vllm/vllm/platforms/cuda.py", line 14, in <module>
    import vllm._C  # noqa
ModuleNotFoundError: No module named 'vllm._C'

How you are installing vllm

pip install -vvv vllm
pip install vllm==0.6.4
pip install -e .

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[cephdon]

The actual error message is:

(vllm) cephdon@ubuntu:~/sources/vllm$ VLLM_USE_PRECOMPILED=1 pip install --editable .
Obtaining file:///mnt/MSAI/home/cephdon/sources/vllm
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build editable ... error
error: subprocess-exited-with-error

× Getting requirements to build editable did not run successfully.
│ exit code: 1
╰─> [20 lines of output]
/tmp/pip-build-env-kav4qhfd/overlay/lib/python3.10/site-packages/torch/_subclasses/functional_tensor.py:295: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:84.)
cpu = _conversion_method_template(device=torch.device("cpu"))
Traceback (most recent call last):
File "/mnt/MSAI/home/cephdon/.local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in
main()
File "/mnt/MSAI/home/cephdon/.local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/mnt/MSAI/home/cephdon/.local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 132, in get_requires_for_build_editable
return hook(config_settings)
File "/tmp/pip-build-env-kav4qhfd/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 483, in get_requires_for_build_editable
return self.get_requires_for_build_wheel(config_settings)
File "/tmp/pip-build-env-kav4qhfd/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 334, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=[])
File "/tmp/pip-build-env-kav4qhfd/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 304, in _get_build_requires
self.run_setup()
File "/tmp/pip-build-env-kav4qhfd/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 320, in run_setup
exec(code, locals())
File "", line 602, in
File "", line 504, in get_vllm_version
RuntimeError: Unknown runtime environment
[end of output]

[cephdon]

(vllm) cephdon@ubuntu:~/sources/vllm$ pip list
Package Version Editable project location

---

absl-py 2.1.0
aiohappyeyeballs 2.4.4
aiohttp 3.11.9
aiosignal 1.3.1
annotated-types 0.7.0
anyio 4.6.2.post1
apache-beam 2.61.0
argcomplete 3.5.1
array_record 0.5.1
astunparse 1.6.3
async-timeout 5.0.1
attrs 24.2.0
beautifulsoup4 4.12.3
blinker 1.9.0
boto 2.49.0
build 1.2.2.post1
cachetools 5.5.0
certifi 2024.8.30
cffi 1.17.1
charset-normalizer 3.4.0
click 8.1.7
cloudpickle 2.2.1
cmake 3.31.1
compressed-tensors 0.8.0
crcmod 1.7
cryptography 43.0.3
cuda-python 12.6.2.post1
datasets 3.1.0
decorator 5.1.1
decorator-args 1.1
dill 0.3.1.1
diskcache 5.6.3
distlib 0.3.9
dm-tree 0.1.8
dnspython 2.7.0
DockerHub-API 0.5
docopt 0.6.2
docstring_parser 0.16
einops 0.8.0
etils 1.11.0
exceptiongroup 1.2.2
fastapi 0.115.6
fastavro 1.9.7
fasteners 0.19
filelock 3.16.1
Flask 3.1.0
flatbuffers 24.3.25
frozenlist 1.5.0
fsspec 2024.9.0
furl 2.1.3
gast 0.5.3
gcs-oauth2-boto-plugin 3.2
gdown 5.2.0
gguf 0.10.0
google-apitools 0.5.32
google-auth 2.17.0
google-auth-httplib2 0.2.0
google-pasta 0.2.0
google-reauth 0.1.1
grpcio 1.65.5
gsutil 5.32
h11 0.14.0
h5py 3.12.1
hatch 1.13.0
hatchling 1.26.3
hdfs 2.7.3
httpcore 1.0.7
httplib2 0.20.4
httptools 0.6.4
httpx 0.28.0
huggingface-hub 0.26.3
hyperlink 21.0.0
idna 3.10
immutabledict 4.2.1
importlib_resources 6.4.5
inflect 7.4.0
iniconfig 2.0.0
interegular 0.3.3
isodate 0.7.2
itsdangerous 2.2.0
Jinja2 3.1.4
jiter 0.8.0
jsonpath-rw 1.4.0
jsonpickle 3.4.2
jsonschema 4.23.0
jsonschema-specifications 2024.10.1
keras 3.7.0
lark 1.2.2
libclang 18.1.1
llvmlite 0.43.0
lm-format-enforcer 0.10.9
Markdown 3.7
markdown-it-py 3.0.0
MarkupSafe 3.0.2
mdurl 0.1.2
mistral_common 1.5.1
ml-dtypes 0.3.2
mlc-llm 0.18.dev74+gd23d6f51 /mnt/MSAI/home/cephdon/sources/mlc-llm/python
mlcroissant 1.0.12
monotonic 1.6
mpmath 1.3.0
msgpack 1.1.0
msgspec 0.18.6
multidict 6.1.0
multiprocess 0.70.9
namex 0.0.8
natsort 8.4.0
nest-asyncio 1.6.0
networkx 3.4.2
ninja 1.11.1.2
numba 0.60.0
numpy 1.26.4
nvidia-cuda-nvrtc-cu12 12.6.85
nvidia-ml-py 12.560.30
oauth2client 4.1.3
objsize 0.7.0
openai 1.56.2
opencv-python-headless 4.10.0.84
opt_einsum 3.4.0
optree 0.13.1
orderedmultidict 1.0.1
orjson 3.10.12
outlines 0.0.46
packaging 24.2
pandas 2.2.3
pandas-stubs 2.2.3.241126
partial-json-parser 0.2.1.1.post4
pathspec 0.12.1
pillow 10.4.0
pip 24.3.1
platformdirs 4.3.6
pluggy 1.5.0
prometheus_client 0.21.1
prometheus-fastapi-instrumentator 7.0.0
promise 2.3
prompt_toolkit 3.0.48
propcache 0.2.1
proto-plus 1.25.0
protobuf 3.20.3
psutil 6.1.0
py-cpuinfo 9.0.0
pyairports 2.1.1
pyarrow 16.1.0
pyarrow-hotfix 0.6
pyasn1 0.6.1
pyasn1_modules 0.4.1
pybind11 2.13.6
pycountry 24.6.1
pycparser 2.22
pydantic 2.10.3
pydantic_core 2.27.1
pydot 1.4.2
Pygments 2.18.0
pymongo 4.10.1
pyOpenSSL 24.2.1
pyparsing 3.2.0
pyproject_hooks 1.2.0
PySocks 1.7.1
pytest 8.3.4
python-dateutil 2.9.0.post0
python-dotenv 1.0.1
pytz 2024.2
pyu2f 0.1.5
PyYAML 6.0.2
pyzmq 26.2.0
ray 2.40.0
rdflib 7.1.1
redis 5.2.1
referencing 0.35.1
regex 2024.11.6
requests 2.32.3
retry-decorator 1.1.1
rich 13.9.4
rpds-py 0.22.3
rsa 4.7.2
safetensors 0.4.5
scipy 1.14.1
sentencepiece 0.2.0
setuptools 75.1.0
setuptools-scm 8.1.0
shellingham 1.5.4
shortuuid 1.0.13
simple-parsing 0.1.6
six 1.17.0
sniffio 1.3.1
sortedcontainers 2.4.0
soupsieve 2.6
starlette 0.41.3
sympy 1.13.1
tabulate 0.9.0
tensorboard 2.16.2
tensorboard-data-server 0.7.2
tensorflow 2.16.1+nv24.8
tensorflow-io-gcs-filesystem 0.37.1
tensorflow-metadata 1.16.1
termcolor 2.5.0
tfds-nightly 4.9.7.dev202412060044
tiktoken 0.7.0
tokenizers 0.20.3
toml 0.10.2
tomli 2.2.1
tomli_w 1.1.0
tomlkit 0.13.2
torch 2.5.1
torch2trt 0.5.0 /mnt/MSAI/home/cephdon/sources/torch2trt
tornado 6.4.2
tqdm 4.67.1
transformers 4.46.3
triton 3.2.0+git07e1cc63 /mnt/MSAI/home/cephdon/sources/triton/python
trove-classifiers 2024.10.21.16
tvm 0.18.dev249+ge6b2a55d1 /mnt/MSAI/home/cephdon/sources/tvm-unity/python
typeguard 4.4.1
types-pytz 2024.2.0.20241003
typing_extensions 4.12.2
tzdata 2024.2
urllib3 2.2.3
userpath 1.9.2
uv 0.5.7
uvicorn 0.32.1
uvloop 0.21.0
virtualenv 20.28.0
watchfiles 1.0.0
wcwidth 0.2.13
websockets 14.1
Werkzeug 3.1.3
wget 3.2
wheel 0.44.0
wrapt 1.17.0
xgrammar 0.1.5 /mnt/MSAI/home/cephdon/sources/xgrammar/python
xxhash 3.5.0
yarl 1.18.3
zstandard 0.23.0

[cephdon]

I should note that this is on an NVIDIA AGX Orin 64gb machine, so aarch64. I have cuda 12.6, but building from source has this same issue that I cannot seem to get past. Installing numpy (any version) does not resolve the issue. I am using miniconda to manage virtual environments. pip list and conda list both show numpy.

Note that the error message above is trying to use the CPU device, even if I have the cuda version of torch installed. Using the CPU version of torch does not resolve the issue.

[shen-shanshan]

I should note that this is on an NVIDIA AGX Orin 64gb machine, so aarch64. I have cuda 12.6, but building from source has this same issue that I cannot seem to get past. Installing numpy (any version) does not resolve the issue. I am using miniconda to manage virtual environments. pip list and conda list both show numpy.

Note that the error message above is trying to use the CPU device, even if I have the cuda version of torch installed. Using the CPU version of torch does not resolve the issue.

I have got the same problem, waiting for some possible solutions. 😢

... UserWarning: Failed to initialize NumPy: No module named 'numpy' ...

[shuguang99]

I have got the same problem, waiting for some possible solutions. 😢

[chris-aeviator]

same problem building from main (ea7bd68)

[DarkLight1337]

@youkaichao @dtrifiro any idea about this?

[dtrifiro]

@cephdon a couple notes:

• since you're using aarch64, you should not be using VLLM_USE_PRECOMPILED, since that uses prebuilt CUDA binaries for x86_64
• The current build setup for non-CUDA builds is broken when using build-isolation (PEP517/518 style builds) so you'll have to do the following for aarch64:

pip install -r requirements-build.txt -r requirements-cuda-arm64.txt
pip install  --no-build-isolation --verbose .

The reason for the error you're getting is that we currenty have torch==2.5.1 in pyproject.toml, so the torch installation when using build isolation is the version provided by pyproject.toml, which is not correct. (requirements-cuda-arm64.txt has 2.6.0.dev.<...> with CUDA 12.4).

There has been some effort in fixing the build process for isolated builds and different targets here, but it's been on hold for a while. Perhaps I'll try do do some more work on it next month.

@chris-aeviator are you using aarch64 as well?

[youkaichao]

@cephdon

... UserWarning: Failed to initialize NumPy: No module named 'numpy' ...

this is a warning, you can ignore.

File "", line 504, in get_vllm_version
RuntimeError: Unknown runtime environment

this is the real error.

see the great note from @dtrifiro

[surak]

@youkaichao @dtrifiro there is no requirements-cuda-arm64.txt on GitHub

[youkaichao]

for building on arm64, please follow https://docs.vllm.ai/en/latest/serving/deploying_with_docker.html#building-for-arm64-aarch64

there is no requirements-cuda-arm64.txt on GitHub

you are right, we deleted it. please follow the up-to-date doc.

[surak]

That is to build in a container, which is thoroughly useless for many use cases.

We can't use it on our supercomputers, we can't create a pip-installable wheel for our users either. If there's an option to build from source, great. If there's an argument to give to pip or to uv pip, even better.

[youkaichao]

That is to build in a container, which is thoroughly useless for many use cases.

We can't use it on our supercomputers, we can't create a pip-installable wheel for our users either. If there's an option to build from source, great. If there's an argument to give to pip or to uv pip, even better.

I don't get it. Building in a container just makes things more isolated, you can of course use it without a container.

We have clear doc on how to build from source on arm GPUs: https://docs.vllm.ai/en/latest/getting_started/installation.html#use-an-existing-pytorch-installation

[surak]

@youkaichao

I don't get it. Building in a container just makes things more isolated, you can of course use it without a container.

That's not how supercomputing centers work. They tend to compile everything to their platform, and try to extract every bit of performance they can. Running a linux kernel and a whole toolchain on top of a a linux kernel and a whole toolchain (repeat as needed) is not acceptable in many places. The overhead is not worth the convenience.

Plus, the security nightmare that docker makes.

I have read your documentation multiple times. I have managed to compile it, but never managed to make it work.

It is got so bad that the market has been offering gh200 under cost because literally no one can run vllm on them properly because you can't bother to offer a precompiled wheel.

Do you need links?

[youkaichao]

because you can't bother to offer a precompiled wheel

we cannot provide wheels, because pytorch does not have official release wheels for gh200 (arm gpus). we have to wait for pytorch 2.6 release.

technically, you can get wheels by copying from the docker container, but the wheel does not specify pytorch as dependency, and you need to install pytorch before installing vllm using that wheel.

I have managed to compile it, but never managed to make it work.

can you elaborate?