NCCL error

[maxmelichov]

I'm trying to load model into LLM(model="meta-llama/Llama-2-7b-chat-hf") and I'm getting the error below

DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/NCCLUtils.hpp:219, invalid argument, NCCL version 2.14.3
ncclInvalidArgument: Invalid value for an argument.
Last error:
Invalid config blocking attribute value -2147483648

[WoosukKwon]

Hi @maxmelichov, in my experience, the error happened when using old version of pytorch. Please make sure to use torch2.1.0+cu121 by running pip install --upgrade torch.

[insunkang]

same issue in cuda 11.8 , torch2.1.0+cu118

[Grey4sh]

same issue in cuda 12.1, torch 2.1.1 + cu121, did u solve it ?

[sanfuyee]

occur the same problem

[NintendoLink]

same error.
torch 2.1.1
torchaudio 2.1.1
torchvision 0.16.1

[sam-iink]

Same issue for me after I updated to 0.2.2 with Pytorch '2.1.1+cu121'.
0.2.1-post1 with '2.0.1+cu11.8' was working.

[jxh4945777]

same issue

[ywglf]

pip list | grep nccl
to check if you have two versions， you should remove the unnecessary one

[jxh4945777]

pip list | grep nccl to check if you have two versions， you should remove the unnecessary one

THX, solved

[noelEOS]

pip list | grep nccl to check if you have two versions， you should remove the unnecessary one

This solved it for me as well. Thanks!

[BeastyZ]

pip list | grep nccl to check if you have two versions， you should remove the unnecessary one

It works for me, too. Thanks!

[goswamig]

pip install --upgrade torch

solved the issue for me.

torch 2.1.2
torchaudio 2.1.2
torchvision 0.16.2

[iseesaw]

pip list | grep nccl to check if you have two versions， you should remove the unnecessary one

Thx, I love you!

[songkq]

Same error in env. Could you please give some advice?

CUDA_VISIBLE_DEVICES="5,7" python3 -m vllm.entrypoints.openai.api_server --model Qwen1.5-72B-Chat-GPTQ-Int4 --port 21002 --gpu-memory-utilization 0.98 --tensor-parallel-size 2

nvidia-nccl-cu12==2.18.1
torch==2.1.2+cu121
vllm==0.3.0 or 0.3.1

File "/workspace/miniconda3/envs/collie/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 217, in <module>
    engine = AsyncLLMEngine.from_engine_args(engine_args)
  File "/workspace/miniconda3/envs/collie/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 623, in from_engine_args
    engine = cls(parallel_config.worker_use_ray,
  File "/workspace/miniconda3/envs/collie/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 319, in __init__
    self.engine = self._init_engine(*args, **kwargs)
  File "/workspace/miniconda3/envs/collie/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 364, in _init_engine
    return engine_class(*args, **kwargs)
  File "/workspace/miniconda3/envs/collie/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 109, in __init__
    self._init_workers_ray(placement_group)
  File "/workspace/miniconda3/envs/collie/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 271, in _init_workers_ray
    self._run_workers("init_model")
  File "/workspace/miniconda3/envs/collie/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 983, in _run_workers
    driver_worker_output = getattr(self.driver_worker,
  File "/workspace/miniconda3/envs/collie/lib/python3.10/site-packages/vllm/worker/worker.py", line 84, in init_model
    init_distributed_environment(self.parallel_config, self.rank,
  File "/workspace/miniconda3/envs/collie/lib/python3.10/site-packages/vllm/worker/worker.py", line 253, in init_distributed_environment
    torch.distributed.all_reduce(torch.zeros(1).cuda())
  File "/workspace/envs/collie/lib/python3.10/site-packages/torch-2.1.2-py3.10-linux-x86_64.egg/torch/distributed/c10d_logger.py", line 47, in wrapper
    return func(*args, **kwargs)
  File "/workspace/miniconda3/envs/collie/lib/python3.10/site-packages/torch-2.1.2-py3.10-linux-x86_64.egg/torch/distributed/distributed_c10d.py", line 2050, in all_reduce
    work = group.allreduce([tensor], opts)
torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/NCCLUtils.hpp:219, invalid argument, NCCL version 2.16.5
ncclInvalidArgument: Invalid value for an argument.

[jinfengfeng]

@songkq I got the same issue. Have you solved it?