[Bug]: assortment of warnings / errors coming out of vllm basic python inference script

[vadimkantorov]

Your current environment

Versions:

>>> import torch; torch.__version__                                                                                                                                       
'2.7.0+cu126'
>>> import transformers; transformers.__version__                                                                                                                         
'4.52.2'
>>> import vllm; vllm.__version__                                                                                                                                         
'0.9.1.dev59+gb6a6e7a52'

🐛 Describe the bug

My script is really basic: preparing model input IDs using tokenizers, constructing vllm.LLM and then invoking .generate(...) once

But I'm somehow getting a bunch of different nasty errors and warnings. Are they expected? Is it possible to eliminate them?

Thanks!

huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
        - Avoid using `tokenizers` before the fork if possible
        - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

[W523 21:36:39.228028808 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=170, addr=[localhost]:60438, remote=[localhost]:50427): failed to recv, got 0 bytes
Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:678 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7f4a777785e8 in /mnt/fs/venv_cu126_py312/lib/python3.12/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x5ba8afe (0x7f4a6065aafe in /mnt/fs/venv_cu126_py312/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x5baae40 (0x7f4a6065ce40 in /mnt/fs/venv_cu126_py312/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x5bab74a (0x7f4a6065d74a in /mnt/fs/venv_cu126_py312/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x2a9 (0x7f4a606571a9 in /mnt/fs/venv_cu126_py312/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #5: c10d::ProcessGroupNCCL::heartbeatMonitor() + 0x379 (0x7f4a1d8509a9 in /mnt/fs/venv_cu126_py312/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
frame #6: <unknown function> + 0xdc253 (0x7f4b1dda3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #7: <unknown function> + 0x94ac3 (0x7f4b238e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #8: <unknown function> + 0x126850 (0x7f4b23973850 in /lib/x86_64-linux-gnu/libc.so.6)

[W523 21:36:40.236201644 ProcessGroupNCCL.cpp:1659] [PG ID 0 PG GUID 0 Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: failed to recv, got 0 bytes
[rank6]:[W523 21:36:40.236202281 ProcessGroupNCCL.cpp:1659] [PG ID 0 PG GUID 0 Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: failed to recv, got 0 bytes

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[sata-bench]

faced same issue upgrading from 0.8.5.post1 to 0.9.1

[shixianc]

+1

[WayenVan]

same issue in 0.9.1 after runing over 6000 iterations

[rlangefe]

I also ran into this. It happened when I used the data parallel mode. Things are fine for 2 or 4 GPUs, but this error pops up when I run with 8 GPUs.

[youkaichao]

Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early)

this usually means some processes died (especially the rank 0 process died), and the pytorch nccl monitor thread keeps printing the error.

you can suppress the error by setting TORCH_NCCL_ENABLE_MONITORING=0 env var. then the log can be cleaner, and you will see the root cause of exception.

[Ucag]

Same problem. I have to rollback to 0.8

[SunGaofeng]

same boat, when I use vllm 0.9.1.

When rollback to 0.8.5, it's ok. But I have to use the newest version.

Can anyone help solve this problem? Why could it work well in old version but cause problems in the new version? Can you fix it?

[yy1213max]

Same issue in 0.9.2；
Firstly, I verified that is fine for 4 GPUs, but with error when I run with 8 GPUs;
Then, after adding the environment variable TORCH_NCCL_ENABLE_MONITORING=0, the following trace information is displayed.

[nlpkiddo-2001]

INFO: Started server process [5887] INFO: Waiting for application startup. 2025-08-29 07:33:23.675792: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0`.
2025-08-29 07:33:23.705373: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-08-29 07:33:23.705424: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-08-29 07:33:23.706581: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-08-29 07:33:23.712176: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
AttributeError: 'MessageFactory' object has no attribute 'GetPrototype'
AttributeError: 'MessageFactory' object has no attribute 'GetPrototype'
AttributeError: 'MessageFactory' object has no attribute 'GetPrototype'
AttributeError: 'MessageFactory' object has no attribute 'GetPrototype'
2025-08-29 07:33:24.436418: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
AttributeError: 'MessageFactory' object has no attribute 'GetPrototype'
INFO 08-29 07:33:25 [init.py:244] Automatically detected platform cuda.
INFO 08-29 07:33:38 [config.py:823] This model supports multiple tasks: {'embed', 'score', 'reward', 'generate', 'classify'}. Defaulting to 'generate'.
INFO 08-29 07:33:39 [arg_utils.py:1653] Engine in background thread is experimental on VLLM_USE_V1=1. Falling back to V0 Engine.
WARNING 08-29 07:33:39 [arg_utils.py:1479] Chunked prefill is enabled by default for models with max_model_len > 32K. Chunked prefill might not work with some features or models. If you encounter any issues, please disable by launching with --enable-chunked-prefill=False.
INFO 08-29 07:33:39 [config.py:1946] Defaulting to use mp for distributed inference
INFO 08-29 07:33:39 [config.py:2195] Chunked prefill is enabled with max_num_batched_tokens=2048.
WARNING 08-29 07:33:39 [cuda.py:91] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
INFO 08-29 07:33:39 [llm_engine.py:230] Initializing a V0 LLM engine (v0.9.1) with config: model='/home/sas/saved/big_models/qwen3_text_30b_a3b_fp8', speculative_config=None, tokenizer='/home/sas/saved/big_models/qwen3_text_30b_a3b_fp8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=fp8, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/home/sas/saved/big_models/qwen3_text_30b_a3b_fp8, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=None, chunked_prefill_enabled=True, use_async_output_proc=False, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":0,"local_cache_dir":null}, use_cached_outputs=False,
WARNING 08-29 07:33:39 [multiproc_worker_utils.py:307] Reducing Torch parallelism from 64 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(VllmWorkerProcess pid=6157) INFO 08-29 07:33:39 [multiproc_worker_utils.py:226] Worker ready; awaiting tasks
INFO 08-29 07:33:39 [cuda.py:327] Using Flash Attention backend.
(VllmWorkerProcess pid=6157) INFO 08-29 07:33:39 [cuda.py:327] Using Flash Attention backend.
INFO 08-29 07:33:41 [utils.py:1126] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=6157) INFO 08-29 07:33:41 [utils.py:1126] Found nccl from library libnccl.so.2
INFO 08-29 07:33:41 [pynccl.py:70] vLLM is using nccl==2.26.2
(VllmWorkerProcess pid=6157) INFO 08-29 07:33:41 [pynccl.py:70] vLLM is using nccl==2.26.2
(VllmWorkerProcess pid=6157) INFO 08-29 07:33:41 [custom_all_reduce_utils.py:246] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_0,1.json
INFO 08-29 07:33:41 [custom_all_reduce_utils.py:246] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_0,1.json
INFO 08-29 07:33:41 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[1], buffer_handle=(1, 4194304, 6, 'psm_e0e5ae7a'), local_subscribe_addr='ipc:///tmp/c18cffc3-248a-46b7-9e8e-4de8328786e7', remote_subscribe_addr=None, remote_addr_ipv6=False)
INFO 08-29 07:33:41 [parallel_state.py:1065] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
(VllmWorkerProcess pid=6157) INFO 08-29 07:33:41 [parallel_state.py:1065] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 1, EP rank 1
INFO 08-29 07:33:41 [model_runner.py:1171] Starting to load model /home/sas/saved/big_models/qwen3_text_30b_a3b_fp8...
(VllmWorkerProcess pid=6157) INFO 08-29 07:33:41 [model_runner.py:1171] Starting to load model /home/sas/saved/big_models/qwen3_text_30b_a3b_fp8...
Loading safetensors checkpoint shards: 0% Completed | 0/7 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 14% Completed | 1/7 [00:00<00:05, 1.05it/s]
Loading safetensors checkpoint shards: 29% Completed | 2/7 [00:01<00:05, 1.01s/it]
Loading safetensors checkpoint shards: 43% Completed | 3/7 [00:02<00:03, 1.02it/s]
Loading safetensors checkpoint shards: 57% Completed | 4/7 [00:03<00:02, 1.22it/s]
Loading safetensors checkpoint shards: 71% Completed | 5/7 [00:04<00:01, 1.11it/s]
Loading safetensors checkpoint shards: 86% Completed | 6/7 [00:05<00:00, 1.03it/s]
Loading safetensors checkpoint shards: 100% Completed | 7/7 [00:06<00:00, 1.04it/s]
Loading safetensors checkpoint shards: 100% Completed | 7/7 [00:06<00:00, 1.06it/s]

INFO 08-29 07:33:49 [default_loader.py:272] Loading weights took 6.70 seconds
(VllmWorkerProcess pid=6157) INFO 08-29 07:33:49 [default_loader.py:272] Loading weights took 6.86 seconds
INFO 08-29 07:33:49 [model_runner.py:1203] Model loading took 14.5406 GiB and 7.119403 seconds
(VllmWorkerProcess pid=6157) INFO 08-29 07:33:50 [model_runner.py:1203] Model loading took 14.5406 GiB and 7.276071 seconds
WARNING 08-29 07:33:51 [fp8_utils.py:526] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /home/ohoz/ohoz/lib/python3.9/site-packages/vllm/model_executor/layers/quantization/utils/configs/N=2560,K=2048,device_name=NVIDIA_L40S,dtype=fp8_w8a8,block_shape=[128,128].json
(VllmWorkerProcess pid=6157) WARNING 08-29 07:33:51 [fp8_utils.py:526] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /home/ohoz/ohoz/lib/python3.9/site-packages/vllm/model_executor/layers/quantization/utils/configs/N=2560,K=2048,device_name=NVIDIA_L40S,dtype=fp8_w8a8,block_shape=[128,128].json
WARNING 08-29 07:33:51 [fp8_utils.py:526] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /home/ohoz/ohoz/lib/python3.9/site-packages/vllm/model_executor/layers/quantization/utils/configs/N=2048,K=2048,device_name=NVIDIA_L40S,dtype=fp8_w8a8,block_shape=[128,128].json
(VllmWorkerProcess pid=6157) WARNING 08-29 07:33:51 [fp8_utils.py:526] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /home/ohoz/ohoz/lib/python3.9/site-packages/vllm/model_executor/layers/quantization/utils/configs/N=2048,K=2048,device_name=NVIDIA_L40S,dtype=fp8_w8a8,block_shape=[128,128].json
WARNING 08-29 07:33:51 [fused_moe.py:683] Using default MoE config. Performance might be sub-optimal! Config file not found at /home/ohoz/ohoz/lib/python3.9/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=384,device_name=NVIDIA_L40S,dtype=fp8_w8a8,block_shape=[128,128].json
(VllmWorkerProcess pid=6157) WARNING 08-29 07:33:51 [fused_moe.py:683] Using default MoE config. Performance might be sub-optimal! Config file not found at /home/ohoz/ohoz/lib/python3.9/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=384,device_name=NVIDIA_L40S,dtype=fp8_w8a8,block_shape=[128,128].json
(VllmWorkerProcess pid=6157) INFO 08-29 07:33:52 [worker.py:294] Memory profiling takes 2.04 seconds
(VllmWorkerProcess pid=6157) INFO 08-29 07:33:52 [worker.py:294] the current vLLM instance can use total_gpu_memory (44.52GiB) x gpu_memory_utilization (0.90) = 40.07GiB
(VllmWorkerProcess pid=6157) INFO 08-29 07:33:52 [worker.py:294] model weights take 14.54GiB; non_torch_memory takes 0.32GiB; PyTorch activation peak memory takes 0.12GiB; the rest of the memory reserved for KV Cache is 25.09GiB.
INFO 08-29 07:33:52 [worker.py:294] Memory profiling takes 2.29 seconds
INFO 08-29 07:33:52 [worker.py:294] the current vLLM instance can use total_gpu_memory (44.52GiB) x gpu_memory_utilization (0.90) = 40.07GiB
INFO 08-29 07:33:52 [worker.py:294] model weights take 14.54GiB; non_torch_memory takes 0.34GiB; PyTorch activation peak memory takes 1.39GiB; the rest of the memory reserved for KV Cache is 23.79GiB.
INFO 08-29 07:33:53 [executor_base.py:113] # cuda blocks: 16243, # CPU blocks: 2730
INFO 08-29 07:33:53 [executor_base.py:118] Maximum concurrency for 40960 tokens per request: 12.69x
/home/ohoz/python3.9.14/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
`

I am using the same vllm 0.9.1. How to resolve this? I am currently using tp_size = 2 and disabled building cuda graphs by enabling enforce eager.

If some one knows the solution. Kindly share here

[nlpkiddo-2001]

It just got stucked there without any error message. I am using 2 L40s machine for this.

[wyn1015]

same question vllm== 0.10.1.1

[nlpkiddo-2001]

Hi @wyn1015 can you tell me your system RAM memory! I had very low memory lets say 10GB once I increased issue was rectified

[wyn1015]

           total        used        free      shared  buff/cache   available

Mem: 1.0Ti 86Gi 300Gi 76Mi 619Gi 914Gi
Swap: 0B 0B 0B 应该是足够的，我替换vllm版本==0.8.5和post版本还是报错

[wyn1015]

Hi @wyn1015 can you tell me your system RAM memory! I had very low memory lets say 10GB once I increased issue was rectified
thanks for your reply，my error as follow：
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x98 (0x7f1e69b785e8 in /conda-envs/py310/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: + 0x5ba8bfe (0x7f1e52ed5bfe in /conda-envs/py310/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #2: + 0x5baa458 (0x7f1e52ed7458 in /conda-envs/py310/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #3: + 0x5babc3e (0x7f1e52ed8c3e in /conda-envs/py310/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x298 (0x7f1e52ed2298 in conda-envs/py310/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #5: c10d::ProcessGroupNCCL::heartbeatMonitor() + 0x379 (0x7f1e145d19f9 in conda-envs/py310/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: + 0xdc253 (0x7f1e73929253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #7: + 0x94ac3 (0x7f1f1a8c0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #8: + 0x126850 (0x7f1f1a952850 in /lib/x86_64-linux-gnu/libc.so.6)