NCCL hanging during inference

[flexwang]

with vllm v0.2.7, I saw the nccl hanging for allreduce:

�[36m(RayWorkerVllm pid=5085)�[0m [E ProcessGroupNCCL.cpp:475] [Rank 4] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=20518, OpType=ALLREDUCE, NumelIn=106496, NumelOut=106496, Timeout(ms)=1800000) ran for 1800270 milliseconds before timing out.

after switching to v0.3.0(with custom all reduce), it's gather

(RayWorkerVllm pid=4775) [E ProcessGroupNCCL.cpp:475] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=369526, OpType=GATHER, NumelIn=4000, NumelOut=0, Timeout(ms)=1800000) ran for 1800252 milliseconds before timing out.

[flexwang]

https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/cudagraph.html

Having multiple outstanding NCCL operations that are any combination of graph-captured or non-captured is supported. There is a caveat that the mechanism NCCL uses internally to accomplish this has been seen to cause CUDA to deadlock when the graphs of multiple communicators are cudaGraphLaunch()’d from the same thread. To disable this mechansim see the environment variable NCCL_GRAPH_MIXING_SUPPORT.

Looks like there is issue between nccl and cuda graph. I use enforce_eager and it seems to be fixed.

[flexwang]

Looks like there is issue between nccl and cuda graph. I use enforce_eager and it seems to be fixed.

More updates, even after disabling cudagraph by setting enforce_eager to true, we still see the nccl hanging issue.

(RayWorkerVllm pid=4850) [E ProcessGroupNCCL.cpp:475] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1494, OpType=GATHER, NumelIn=76000, NumelOut=0, Timeout(ms)=1800000) ran for 1800855 milliseconds before timing out.

[NikolaBorisov]

I reported the same issue in #2731. you need enforce_eager and disable_custom_all_reduce to fix it. But the real issue is how to fix the cuda_graphs and custom all reduce to make it work.

[WoosukKwon]

Closed as #2811 fixes this. Please feel free to re-open the issue if you find the bug persists.

[LokiLiu]

@WoosukKwon I met same problem, but after using PyTorch 2.2.0, this problem has been resolved.

[WangErXiao]

@WoosukKwon I met same problem, but after using PyTorch 2.2.0, this problem has been resolved.

hi, do you use 0.2.7 or 0.3.0?