[Installation]: Nvidia runtime issue? On new VLLM 0.7.0

[Playerrrrr]

Your current environment

The output of `python collect_env.py`

docker run --runtime nvidia --gpus all -v ~/.cache/huggingface:/root/.cache/huggingface -p 8000:8000 --ipc=host -e VLLM_ENABLE_PREFIX_CACHING=true --name qwen2.5_20250128 vllm/vllm-openai:v0.7.0 --model Qwen/Qwen2.5-72B-Instruct --tensor-parallel-size=4 --gpu-memory-utilization=0.90 --enforce-eager --rope-scaling '{"type": "yarn","factor": 4,"original_max_position_embeddings": 32768}'
error:
/usr/bin/ld: cannot find -lcuda: No such file or directory

How you are installing vllm

docker

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[jamesbraza]

I hit this as well during vllm serve --tensor-parallel-size 2 today with vllm==0.7.0:

INFO 01-28 13:54:38 weight_utils.py:251] Using model weights format ['*.safetensors']
(VllmWorkerProcess pid=1241396) INFO 01-28 13:54:38 weight_utils.py:251] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:00<00:00,  1.48it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00,  1.14it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00,  1.18it/s]

INFO 01-28 13:54:40 model_runner.py:1115] Loading model weights took 7.1441 GB
(VllmWorkerProcess pid=1241396) INFO 01-28 13:54:41 model_runner.py:1115] Loading model weights took 7.1441 GB
/usr/bin/ld: cannot find -lcuda: No such file or directory
collect2: error: ld returned 1 exit status
/usr/bin/ld: cannot find -lcuda: No such file or directory
collect2: error: ld returned 1 exit status
/usr/bin/ld: cannot find -lcuda: No such file or directory
/usr/bin/ld: cannot find -lcuda: No such file or directory
collect2: error: ld returned 1 exit status
collect2: error: ld returned 1 exit status

[mgoin]

@tlrmchlsmth would you have an idea? This seems related to #12424

[robertgshaw2-redhat]

@russellb this looks similar to what you were helping dan with

[tlrmchlsmth]

@tlrmchlsmth would you have an idea? This seems related to #12424

Yep, does seem suspicious. Not sure what's going wrong though

[tlrmchlsmth]

We might need to do something like this instead:
https://github.com/pytorch/pytorch/blob/50f834f13417e4e4b930472c8cf237aa5a60f38e/c10/cuda/driver_api.cpp#L12

[dhuangnm]

I hit similar issue on my build instance (Ubuntu 20.04) and here is what i did to workaround the error:

1. find out where libcuda.so is installed on the instance, e.g. on my machine with CUDA 12.4 installed, it's located under:
   /usr/local/cuda-12.4/targets/x86_64-linux/lib/stubs/libcuda.so

2. softlink /usr/lib64/libcuda.so to the libcuda.so you found above:
   sudo ln -s /usr/local/cuda-12.4/targets/x86_64-linux/lib/stubs/libcuda.so /usr/lib64/libcuda.so

The ld command looks only looking for libraries under certain locations. Since the libcuda.so is not under where it's looking for thus the error. After setting the softlink, vllm can build and run successfully.

[tlrmchlsmth]

I've put up #12552 to revert #12424.

For those having issues with vLLM 0.7.0, the easiest solution will be adding the directory containing libcuda.so to your LD_LIBRARY_PATH environment variable

[dhuangnm]

I tried setting LD_LIBRARY_PATH initially but it didn't work for me for some reason. The ld command still complained about -lcuda not found and I had to use the softlink.

[gargnipungarg]

+1

[mgoin]

This should have been fixed with #12552 so please wait for the next release to include the revert

[stefanobranco]

I assume this only happens for 0.7.0 for everyone here then, since the reverted change is relatively recent? I'm asking because I've been having this issue ever since 0.6.5 which would suggest a different root cause (or a different issue altogether) as also mentioned here #11643