[Usage]: openbmb-MiniCPM-Llama3-V-2_5 在vllm上无法多gpu使用

[renjingneng]

Your current environment

报错：RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! (when checking argument for argument weight in method wrapper_CUDA__cudnn_convolution)

(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] File "/root/miniconda3/envs/minicpmv/lib/python3.10/site-packages/transformers/models/idefics2/modeling_idefics2.py", line 617, in forward
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] hidden_states = self.embeddings(pixel_values=pixel_values, patch_attention_mask=patch_attention_mask)
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] File "/root/miniconda3/envs/minicpmv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] File "/root/miniconda3/envs/minicpmv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] File "/root/miniconda3/envs/minicpmv/lib/python3.10/site-packages/transformers/models/idefics2/modeling_idefics2.py", line 162, in forward
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] patch_embeds = self.patch_embedding(pixel_values)
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] File "/root/miniconda3/envs/minicpmv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] File "/root/miniconda3/envs/minicpmv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] File "/root/miniconda3/envs/minicpmv/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 460, in forward
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] return self._conv_forward(input, self.weight, self.bias)
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] File "/root/miniconda3/envs/minicpmv/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 456, in _conv_forward
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] return F.conv2d(input, weight, bias, self.stride,
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226] RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! (when checking argument for argument weight in method wrapper_CUDA__cudnn_convolution)
(VllmWorkerProcess pid=1470366) ERROR 07-26 13:31:20 multiproc_worker_utils.py:226]
^C^C^C^C^C/root/miniconda3/envs/minicpmv/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
Killed

How would you like to use vllm

No response

[renjingneng]

补充：这个命令 vllm serve /aixunlian/renjingneng/tasks/task_6/model/openbmb-MiniCPM-Llama3-V-2_5 --port 8010 --trust-remote-code --tensor-parallel-size 2 --max_model_len 2000 报错

[HwwwwwwwH]

补充：这个命令 vllm serve /aixunlian/renjingneng/tasks/task_6/model/openbmb-MiniCPM-Llama3-V-2_5 --port 8010 --trust-remote-code --tensor-parallel-size 2 --max_model_len 2000 报错

I'll check this. BTW, is this issue #9 still exists?

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!