[Feature]: Qwen3 Omni Support

[VivekMalipatel]

🚀 The feature, motivation and pitch

I am encountering this when deploying Qwen3 Omni on H200

[service]: INFO 09-27 12:03:27 [init.py:216] Automatically detected platform cuda.
[service]: �[1;36m(APIServer pid=20)�[0;0m INFO 09-27 12:03:35 [api_server.py:1839] vLLM API server version 0.11.0rc2.dev33+gc216119d6
[service]: �[1;36m(APIServer pid=20)�[0;0m INFO 09-27 12:03:35 [utils.py:233] non-default args: {'model_tag': 'Qwen/Qwen3-Omni-30B-A3B-Instruct', 'host': '0.0.0.0', 'port': 8001, 'api_key': ['test-key'], 'model': 'Qwen/Qwen3-Omni-30B-A3B-Instruct', 'trust_remote_code': True, 'enforce_eager': True, 'gpu_memory_utilization': 0.85, 'enable_prefix_caching': True}
[service]: �[1;36m(APIServer pid=20)�[0;0m The argument trust_remote_code is to be used with Auto classes. It has no effect here and is ignored.
[service]: �[1;36m(APIServer pid=20)�[0;0m Traceback (most recent call last):
[service]: �[1;36m(APIServer pid=20)�[0;0m File "/usr/local/bin/vllm", line 10, in
[service]: �[1;36m(APIServer pid=20)�[0;0m sys.exit(main())
[service]: �[1;36m(APIServer pid=20)�[0;0m ^^^^^^
[service]: �[1;36m(APIServer pid=20)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 54, in main
[service]: �[1;36m(APIServer pid=20)�[0;0m args.dispatch_function(args)
[service]: �[1;36m(APIServer pid=20)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 57, in cmd
[service]: �[1;36m(APIServer pid=20)�[0;0m uvloop.run(run_server(args))
[service]: �[1;36m(APIServer pid=20)�[0;0m File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 109, in run
[service]: �[1;36m(APIServer pid=20)�[0;0m return __asyncio.run(
[service]: �[1;36m(APIServer pid=20)�[0;0m ^^^^^^^^^^^^^^
[service]: �[1;36m(APIServer pid=20)�[0;0m File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
[service]: �[1;36m(APIServer pid=20)�[0;0m return runner.run(main)
[service]: �[1;36m(APIServer pid=20)�[0;0m ^^^^^^^^^^^^^^^^
[service]: �[1;36m(APIServer pid=20)�[0;0m File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
[service]: �[1;36m(APIServer pid=20)�[0;0m return self._loop.run_until_complete(task)
[service]: �[1;36m(APIServer pid=20)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(APIServer pid=20)�[0;0m File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
[service]: �[1;36m(APIServer pid=20)�[0;0m File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 61, in wrapper
[service]: �[1;36m(APIServer pid=20)�[0;0m return await main
[service]: �[1;36m(APIServer pid=20)�[0;0m ^^^^^^^^^^
[service]: �[1;36m(APIServer pid=20)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1884, in run_server
[service]: �[1;36m(APIServer pid=20)�[0;0m await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
[service]: �[1;36m(APIServer pid=20)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1902, in run_server_worker
[service]: �[1;36m(APIServer pid=20)�[0;0m async with build_async_engine_client(
[service]: �[1;36m(APIServer pid=20)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(APIServer pid=20)�[0;0m File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
[service]: �[1;36m(APIServer pid=20)�[0;0m return await anext(self.gen)
[service]: �[1;36m(APIServer pid=20)�[0;0m ^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(APIServer pid=20)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 180, in build_async_engine_client
[service]: �[1;36m(APIServer pid=20)�[0;0m async with build_async_engine_client_from_engine_args(
[service]: �[1;36m(APIServer pid=20)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(APIServer pid=20)�[0;0m File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
[service]: �[1;36m(APIServer pid=20)�[0;0m return await anext(self.gen)
[service]: �[1;36m(APIServer pid=20)�[0;0m ^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(APIServer pid=20)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 206, in build_async_engine_client_from_engine_args
[service]: �[1;36m(APIServer pid=20)�[0;0m vllm_config = engine_args.create_engine_config(usage_context=usage_context)
[service]: �[1;36m(APIServer pid=20)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(APIServer pid=20)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 1142, in create_engine_config
[service]: �[1;36m(APIServer pid=20)�[0;0m model_config = self.create_model_config()
[service]: �[1;36m(APIServer pid=20)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(APIServer pid=20)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 994, in create_model_config
[service]: �[1;36m(APIServer pid=20)�[0;0m return ModelConfig(
[service]: �[1;36m(APIServer pid=20)�[0;0m ^^^^^^^^^^^^
[service]: �[1;36m(APIServer pid=20)�[0;0m File "/usr/local/lib/python3.12/dist-packages/pydantic/_internal/_dataclasses.py", line 123, in init
[service]: �[1;36m(APIServer pid=20)�[0;0m s.pydantic_validator.validate_python(ArgsKwargs(args, kwargs), self_instance=s)
[service]: �[1;36m(APIServer pid=20)�[0;0m pydantic_core._pydantic_core.ValidationError: 1 validation error for ModelConfig
[service]: �[1;36m(APIServer pid=20)�[0;0m Value error, The checkpoint you are trying to load has model type qwen3_omni_moe but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
[service]: �[1;36m(APIServer pid=20)�[0;0m
[service]: �[1;36m(APIServer pid=20)�[0;0m You can update Transformers with the command pip install --upgrade transformers. If this does not work, and the checkpoint is very new, then there may not be a release version that supports this model yet. In this case, you can get the most up-to-date code by installing Transformers from source with the command pip install git+https://github.com/huggingface/transformers.git [type=value_error, input_value=ArgsKwargs((), {'model': ...rocessor_plugin': None}), input_type=ArgsKwargs]
[service]: �[1;36m(APIServer pid=20)�[0;0m For further information visit https://errors.pydantic.dev/2.11/v/value_error

docker stop vllm-qwen3-omni && docker rm vllm-qwen3-omni
export VLLM_COMMIT=c0ec81836fd47492a900a2538dea461619122555
docker run -d --name vllm-qwen3-omni
--restart unless-stopped
--network host
--gpus all
-v "$HF_CACHE":/root/.cache/huggingface
-e NCCL_P2P_DISABLE=1
public.ecr.aws/q9t5s3a7/vllm-ci-postmerge-repo:${VLLM_COMMIT}
bash -lc '
pip install --no-cache-dir flash-attn
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct
--host 0.0.0.0 --port 8001
--served-model-name o3
--trust-remote-code
--enable-auto-tool-choice
--max-model-len 65536
--gpu-memory-utilization 0.85
--api-key test-key'
docker logs -f vllm-qwen3

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[ywang96]

It already says in the error message
You can update Transformers with the command pip install --upgrade transformers. If this does not work, and the checkpoint is very new, then there may not be a release version that supports this model yet. In this case, you can get the most up-to-date code by installing Transformers from source with the command pip install git+https://github.com/huggingface/transformers.git

Can you try installing the latest transformers main branch?

[VivekMalipatel]

[service]: Collecting git+https://github.com/huggingface/transformers.git
[service]: Cloning https://github.com/huggingface/transformers.git to /tmp/pip-req-build-bq438tso
[service]: Running command git clone --filter=blob:none --quiet https://github.com/huggingface/transformers.git /tmp/pip-req-build-bq438tso
[service]: Resolved https://github.com/huggingface/transformers.git to commit 3e975acc8bf6d029ec0a54b1c5d0691489dfb051
[service]: Installing build dependencies: started
[service]: Installing build dependencies: finished with status 'done'
[service]: Getting requirements to build wheel: started
[service]: Getting requirements to build wheel: finished with status 'done'
[service]: Preparing metadata (pyproject.toml): started
[service]: Preparing metadata (pyproject.toml): finished with status 'done'
[service]: Requirement already satisfied: filelock in /usr/local/lib/python3.12/dist-packages (from transformers==4.57.0.dev0) (3.19.1)
[service]: Collecting huggingface-hub==1.0.0.rc2 (from transformers==4.57.0.dev0)
[service]: Downloading huggingface_hub-1.0.0rc2-py3-none-any.whl.metadata (14 kB)
[service]: Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.12/dist-packages (from transformers==4.57.0.dev0) (2.2.6)
[service]: Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.12/dist-packages (from transformers==4.57.0.dev0) (25.0)
[service]: Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.12/dist-packages (from transformers==4.57.0.dev0) (6.0.2)
[service]: Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.12/dist-packages (from transformers==4.57.0.dev0) (2025.9.1)
[service]: Requirement already satisfied: requests in /usr/local/lib/python3.12/dist-packages (from transformers==4.57.0.dev0) (2.32.5)
[service]: Requirement already satisfied: tokenizers<=0.23.0,>=0.22.0 in /usr/local/lib/python3.12/dist-packages (from transformers==4.57.0.dev0) (0.22.0)
[service]: Requirement already satisfied: safetensors>=0.4.3 in /usr/local/lib/python3.12/dist-packages (from transformers==4.57.0.dev0) (0.6.2)
[service]: Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.12/dist-packages (from transformers==4.57.0.dev0) (4.67.1)
[service]: Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.12/dist-packages (from huggingface-hub==1.0.0.rc2->transformers==4.57.0.dev0) (2025.9.0)
[service]: Requirement already satisfied: httpx<1,>=0.23.0 in /usr/local/lib/python3.12/dist-packages (from huggingface-hub==1.0.0.rc2->transformers==4.57.0.dev0) (0.28.1)
[service]: Collecting typer-slim (from huggingface-hub==1.0.0.rc2->transformers==4.57.0.dev0)
[service]: Downloading typer_slim-0.19.2-py3-none-any.whl.metadata (16 kB)
[service]: Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.12/dist-packages (from huggingface-hub==1.0.0.rc2->transformers==4.57.0.dev0) (4.15.0)
[service]: Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in /usr/local/lib/python3.12/dist-packages (from huggingface-hub==1.0.0.rc2->transformers==4.57.0.dev0) (1.1.10)
[service]: Requirement already satisfied: anyio in /usr/local/lib/python3.12/dist-packages (from httpx<1,>=0.23.0->huggingface-hub==1.0.0.rc2->transformers==4.57.0.dev0) (4.10.0)
[service]: Requirement already satisfied: certifi in /usr/local/lib/python3.12/dist-packages (from httpx<1,>=0.23.0->huggingface-hub==1.0.0.rc2->transformers==4.57.0.dev0) (2025.8.3)
[service]: Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.12/dist-packages (from httpx<1,>=0.23.0->huggingface-hub==1.0.0.rc2->transformers==4.57.0.dev0) (1.0.9)
[service]: Requirement already satisfied: idna in /usr/local/lib/python3.12/dist-packages (from httpx<1,>=0.23.0->huggingface-hub==1.0.0.rc2->transformers==4.57.0.dev0) (3.10)
[service]: Requirement already satisfied: h11>=0.16 in /usr/local/lib/python3.12/dist-packages (from httpcore==1.*->httpx<1,>=0.23.0->huggingface-hub==1.0.0.rc2->transformers==4.57.0.dev0) (0.16.0)
[service]: INFO: pip is looking at multiple versions of tokenizers to determine which version is compatible with other requirements. This could take a while.
[service]: Collecting tokenizers<=0.23.0,>=0.22.0 (from transformers==4.57.0.dev0)
[service]: Downloading tokenizers-0.22.1-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.8 kB)
[service]: Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.12/dist-packages (from anyio->httpx<1,>=0.23.0->huggingface-hub==1.0.0.rc2->transformers==4.57.0.dev0) (1.3.1)
[service]: Requirement already satisfied: charset_normalizer<4,>=2 in /usr/local/lib/python3.12/dist-packages (from requests->transformers==4.57.0.dev0) (3.4.3)
[service]: Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.12/dist-packages (from requests->transformers==4.57.0.dev0) (2.5.0)
[service]: Requirement already satisfied: click>=8.0.0 in /usr/local/lib/python3.12/dist-packages (from typer-slim->huggingface-hub==1.0.0.rc2->transformers==4.57.0.dev0) (8.2.1)
[service]: Downloading huggingface_hub-1.0.0rc2-py3-none-any.whl (528 kB)
[service]: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 528.7/528.7 kB 30.3 MB/s 0:00:00
[service]: Downloading tokenizers-0.22.1-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.3 MB)
[service]: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 MB 95.8 MB/s 0:00:00
[service]: Downloading typer_slim-0.19.2-py3-none-any.whl (46 kB)
[service]: Building wheels for collected packages: transformers
[service]: Building wheel for transformers (pyproject.toml): started
[service]: Building wheel for transformers (pyproject.toml): finished with status 'done'
[service]: Created wheel for transformers: filename=transformers-4.57.0.dev0-py3-none-any.whl size=11575752 sha256=99dc4be7a86789347195e95e8dbb49545ce7fd284a2d73640b3ae6449df3d3f3
[service]: Stored in directory: /tmp/pip-ephem-wheel-cache-p9bo1xth/wheels/54/cb/3f/83103de5575c534436d6a4686686dead458238dfaf1147e98d
[service]: Successfully built transformers
[service]: Installing collected packages: typer-slim, huggingface-hub, tokenizers, transformers
[service]: Attempting uninstall: huggingface-hub
[service]: Found existing installation: huggingface-hub 0.34.4
[service]: Uninstalling huggingface-hub-0.34.4:
[service]: Successfully uninstalled huggingface-hub-0.34.4
[service]: Attempting uninstall: tokenizers
[service]: Found existing installation: tokenizers 0.22.0
[service]: Uninstalling tokenizers-0.22.0:
[service]: Successfully uninstalled tokenizers-0.22.0
[service]: Attempting uninstall: transformers
[service]: Found existing installation: transformers 4.56.1
[service]: Uninstalling transformers-4.56.1:
[service]: Successfully uninstalled transformers-4.56.1
[service]:
[service]: Successfully installed huggingface-hub-1.0.0rc2 tokenizers-0.22.1 transformers-4.57.0.dev0 typer-slim-0.19.2
[service]: WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
[service]: /usr/local/lib/python3.12/dist-packages/torch/cuda/init.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
[service]: import pynvml # type: ignore[import]
[service]: INFO 09-29 10:51:27 [init.py:216] Automatically detected platform cuda.
[service]: �[1;36m(APIServer pid=7)�[0;0m INFO 09-29 10:51:31 [api_server.py:1896] vLLM API server version 0.10.2
[service]: �[1;36m(APIServer pid=7)�[0;0m INFO 09-29 10:51:31 [utils.py:328] non-default args: {'model_tag': 'Qwen/Qwen3-Omni-30B-A3B-Thinking', 'host': '0.0.0.0', 'port': 8001, 'model': 'Qwen/Qwen3-Omni-30B-A3B-Thinking', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_model_len': 32768, 'gpu_memory_utilization': 0.85}
[service]: �[1;36m(APIServer pid=7)�[0;0m The argument trust_remote_code is to be used with Auto classes. It has no effect here and is ignored.
[service]: �[1;36m(APIServer pid=7)�[0;0m Unrecognized keys in rope_scaling for 'rope_type'='default': {'interleaved', 'mrope_interleaved', 'mrope_section'}
[service]: �[1;36m(APIServer pid=7)�[0;0m INFO 09-29 10:51:37 [init.py:742] Resolved architecture: TransformersForMultimodalLM
[service]: �[1;36m(APIServer pid=7)�[0;0m torch_dtype is deprecated! Use dtype instead!
[service]: �[1;36m(APIServer pid=7)�[0;0m INFO 09-29 10:51:37 [init.py:1815] Using max model len 32768
[service]: �[1;36m(APIServer pid=7)�[0;0m Unrecognized keys in rope_scaling for 'rope_type'='default': {'interleaved', 'mrope_interleaved', 'mrope_section'}
[service]: �[1;36m(APIServer pid=7)�[0;0m INFO 09-29 10:51:38 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=8192.
[service]: �[1;36m(APIServer pid=7)�[0;0m The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set TRANSFORMERS_VERBOSITY=info for more details.
[service]: �[1;36m(APIServer pid=7)�[0;0m WARNING 09-29 10:51:40 [utils.py:181] TransformersForMultimodalLM has no vLLM implementation, falling back to Transformers implementation. Some features may not be supported and performance may not be optimal.
[service]: /usr/local/lib/python3.12/dist-packages/torch/cuda/init.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
[service]: import pynvml # type: ignore[import]
[service]: INFO 09-29 10:51:43 [init.py:216] Automatically detected platform cuda.
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m INFO 09-29 10:51:45 [core.py:654] Waiting for init message from front-end.
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m INFO 09-29 10:51:45 [core.py:76] Initializing a V1 LLM engine (v0.10.2) with config: model='Qwen/Qwen3-Omni-30B-A3B-Thinking', speculative_config=None, tokenizer='Qwen/Qwen3-Omni-30B-A3B-Thinking', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Qwen/Qwen3-Omni-30B-A3B-Thinking, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2","vllm.mamba_mixer","vllm.short_conv","vllm.linear_attention","vllm.plamo2_mamba_mixer","vllm.gdn_attention"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"cudagraph_mode":1,"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"pass_config":{},"max_capture_size":512,"local_cache_dir":null}
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m W0929 10:51:45.369000 346 torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m W0929 10:51:45.369000 346 torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[service]: [W929 10:51:46.139612647 ProcessGroupNCCL.cpp:981] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator())
[service]: [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[service]: [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[service]: [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[service]: [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[service]: [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[service]: [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m INFO 09-29 10:51:46 [parallel_state.py:1165] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m WARNING 09-29 10:51:46 [utils.py:181] TransformersForMultimodalLM has no vLLM implementation, falling back to Transformers implementation. Some features may not be supported and performance may not be optimal.
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m INFO 09-29 10:51:47 [topk_topp_sampler.py:58] Using FlashInfer for top-p & top-k sampling.
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m The image processor of type Qwen2VLImageProcessor is now loaded as a fast processor by default, even if the model checkpoint was saved with a slow processor. This is a breaking change and may produce slightly different outputs. To continue using the slow processor, instantiate this class with use_fast=False. Note that this behavior will be extended to all models in a future release.
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] EngineCore failed to start.
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] Traceback (most recent call last):
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m Process EngineCore_DP0:
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 709, in run_engine_core
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] engine_core = EngineCoreProc(*args, **kwargs)
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 505, in init
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] super().init(vllm_config, executor_class, log_stats,
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 82, in init
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] self.model_executor = executor_class(vllm_config)
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 54, in init
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] self._init_executor()
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 48, in _init_executor
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] self.collective_rpc("init_device")
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] answer = run_method(self.driver_worker, method, args, kwargs)
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/init.py", line 3060, in run_method
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] return func(*args, **kwargs)
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] ^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 611, in init_device
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] self.worker.init_device() # type: ignore
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] ^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 201, in init_device
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] self.model_runner: GPUModelRunner = GPUModelRunner(
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] ^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 383, in init
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] self.mm_budget = MultiModalBudget(
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] ^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/utils.py", line 48, in init
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] .get_max_tokens_per_item_by_nonzero_modality(model_config,
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] File "/usr/local/lib/python3.12/dist-packages/vllm/multimodal/registry.py", line 168, in get_max_tokens_per_item_by_nonzero_modality
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] max_tokens_per_item = self.get_max_tokens_per_item_by_modality(
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] File "/usr/local/lib/python3.12/dist-packages/vllm/multimodal/registry.py", line 144, in get_max_tokens_per_item_by_modality
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] return profiler.get_mm_max_contiguous_tokens(
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] File "/usr/local/lib/python3.12/dist-packages/vllm/multimodal/profiling.py", line 311, in get_mm_max_contiguous_tokens
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] return self._get_mm_max_tokens(seq_len,
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] File "/usr/local/lib/python3.12/dist-packages/vllm/multimodal/profiling.py", line 268, in _get_mm_max_tokens
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] max_tokens_per_item = self.processing_info.get_mm_max_tokens_per_item(
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/transformers.py", line 227, in get_mm_max_tokens_per_item
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] return {"image": self.get_max_image_tokens()}
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/transformers.py", line 233, in get_max_image_tokens
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] mm_tokens = processor._get_num_multimodal_tokens(
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ERROR 09-29 10:51:52 [core.py:718] AttributeError: 'Qwen3OmniMoeProcessor' object has no attribute '_get_num_multimodal_tokens'
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m Traceback (most recent call last):
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m self.run()
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m self._target(*self._args, **self._kwargs)
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 722, in run_engine_core
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m raise e
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 709, in run_engine_core
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m engine_core = EngineCoreProc(*args, **kwargs)
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 505, in init
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m super().init(vllm_config, executor_class, log_stats,
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 82, in init
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m self.model_executor = executor_class(vllm_config)
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 54, in init
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m self._init_executor()
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 48, in _init_executor
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m self.collective_rpc("init_device")
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m answer = run_method(self.driver_worker, method, args, kwargs)
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/utils/init.py", line 3060, in run_method
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m return func(*args, **kwargs)
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 611, in init_device
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m self.worker.init_device() # type: ignore
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 201, in init_device
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m self.model_runner: GPUModelRunner = GPUModelRunner(
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 383, in init
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m self.mm_budget = MultiModalBudget(
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/utils.py", line 48, in init
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m .get_max_tokens_per_item_by_nonzero_modality(model_config,
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/multimodal/registry.py", line 168, in get_max_tokens_per_item_by_nonzero_modality
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m max_tokens_per_item = self.get_max_tokens_per_item_by_modality(
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/multimodal/registry.py", line 144, in get_max_tokens_per_item_by_modality
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m return profiler.get_mm_max_contiguous_tokens(
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/multimodal/profiling.py", line 311, in get_mm_max_contiguous_tokens
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m return self._get_mm_max_tokens(seq_len,
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/multimodal/profiling.py", line 268, in _get_mm_max_tokens
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m max_tokens_per_item = self.processing_info.get_mm_max_tokens_per_item(
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/transformers.py", line 227, in get_mm_max_tokens_per_item
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m return {"image": self.get_max_image_tokens()}
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/transformers.py", line 233, in get_max_image_tokens
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m mm_tokens = processor._get_num_multimodal_tokens(
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m AttributeError: 'Qwen3OmniMoeProcessor' object has no attribute '_get_num_multimodal_tokens'
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m Exception ignored in: <function ExecutorBase.del at 0x773383df2980>
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m Traceback (most recent call last):
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 237, in del
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m self.shutdown()
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 76, in shutdown
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m worker.shutdown()
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 528, in shutdown
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m self.worker.shutdown()
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 675, in shutdown
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m self.model_runner.ensure_kv_transfer_shutdown()
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(EngineCore_DP0 pid=346)�[0;0m AttributeError: 'NoneType' object has no attribute 'ensure_kv_transfer_shutdown'
[service]: [rank0]:[W929 10:51:52.382410051 ProcessGroupNCCL.cpp:1538] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
[service]: �[1;36m(APIServer pid=7)�[0;0m Traceback (most recent call last):
[service]: �[1;36m(APIServer pid=7)�[0;0m File "/usr/local/bin/vllm", line 10, in
[service]: �[1;36m(APIServer pid=7)�[0;0m sys.exit(main())
[service]: �[1;36m(APIServer pid=7)�[0;0m ^^^^^^
[service]: �[1;36m(APIServer pid=7)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 54, in main
[service]: �[1;36m(APIServer pid=7)�[0;0m args.dispatch_function(args)
[service]: �[1;36m(APIServer pid=7)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 50, in cmd
[service]: �[1;36m(APIServer pid=7)�[0;0m uvloop.run(run_server(args))
[service]: �[1;36m(APIServer pid=7)�[0;0m File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 109, in run
[service]: �[1;36m(APIServer pid=7)�[0;0m return __asyncio.run(
[service]: �[1;36m(APIServer pid=7)�[0;0m ^^^^^^^^^^^^^^
[service]: �[1;36m(APIServer pid=7)�[0;0m File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
[service]: �[1;36m(APIServer pid=7)�[0;0m return runner.run(main)
[service]: �[1;36m(APIServer pid=7)�[0;0m ^^^^^^^^^^^^^^^^
[service]: �[1;36m(APIServer pid=7)�[0;0m File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
[service]: �[1;36m(APIServer pid=7)�[0;0m return self._loop.run_until_complete(task)
[service]: �[1;36m(APIServer pid=7)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(APIServer pid=7)�[0;0m File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
[service]: �[1;36m(APIServer pid=7)�[0;0m File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 61, in wrapper
[service]: �[1;36m(APIServer pid=7)�[0;0m return await main
[service]: �[1;36m(APIServer pid=7)�[0;0m ^^^^^^^^^^
[service]: �[1;36m(APIServer pid=7)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1941, in run_server
[service]: �[1;36m(APIServer pid=7)�[0;0m await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
[service]: �[1;36m(APIServer pid=7)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1961, in run_server_worker
[service]: �[1;36m(APIServer pid=7)�[0;0m async with build_async_engine_client(
[service]: �[1;36m(APIServer pid=7)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(APIServer pid=7)�[0;0m File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
[service]: �[1;36m(APIServer pid=7)�[0;0m return await anext(self.gen)
[service]: �[1;36m(APIServer pid=7)�[0;0m ^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(APIServer pid=7)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 179, in build_async_engine_client
[service]: �[1;36m(APIServer pid=7)�[0;0m async with build_async_engine_client_from_engine_args(
[service]: �[1;36m(APIServer pid=7)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(APIServer pid=7)�[0;0m File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
[service]: �[1;36m(APIServer pid=7)�[0;0m return await anext(self.gen)
[service]: �[1;36m(APIServer pid=7)�[0;0m ^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(APIServer pid=7)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 221, in build_async_engine_client_from_engine_args
[service]: �[1;36m(APIServer pid=7)�[0;0m async_llm = AsyncLLM.from_vllm_config(
[service]: �[1;36m(APIServer pid=7)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(APIServer pid=7)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/utils/init.py", line 1589, in inner
[service]: �[1;36m(APIServer pid=7)�[0;0m return fn(*args, **kwargs)
[service]: �[1;36m(APIServer pid=7)�[0;0m ^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(APIServer pid=7)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 212, in from_vllm_config
[service]: �[1;36m(APIServer pid=7)�[0;0m return cls(
[service]: �[1;36m(APIServer pid=7)�[0;0m ^^^^
[service]: �[1;36m(APIServer pid=7)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 136, in init
[service]: �[1;36m(APIServer pid=7)�[0;0m self.engine_core = EngineCoreClient.make_async_mp_client(
[service]: �[1;36m(APIServer pid=7)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(APIServer pid=7)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 102, in make_async_mp_client
[service]: �[1;36m(APIServer pid=7)�[0;0m return AsyncMPClient(*client_args)
[service]: �[1;36m(APIServer pid=7)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(APIServer pid=7)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 769, in init
[service]: �[1;36m(APIServer pid=7)�[0;0m super().init(
[service]: �[1;36m(APIServer pid=7)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 448, in init
[service]: �[1;36m(APIServer pid=7)�[0;0m with launch_core_engines(vllm_config, executor_class,
[service]: �[1;36m(APIServer pid=7)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[service]: �[1;36m(APIServer pid=7)�[0;0m File "/usr/lib/python3.12/contextlib.py", line 144, in exit
[service]: �[1;36m(APIServer pid=7)�[0;0m next(self.gen)
[service]: �[1;36m(APIServer pid=7)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 729, in launch_core_engines
[service]: �[1;36m(APIServer pid=7)�[0;0m wait_for_engine_startup(
[service]: �[1;36m(APIServer pid=7)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 782, in wait_for_engine_startup
[service]: �[1;36m(APIServer pid=7)�[0;0m raise RuntimeError("Engine core initialization failed. "
[service]: �[1;36m(APIServer pid=7)�[0;0m RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

    pip install --no-cache-dir git+https://github.com/huggingface/transformers.git
    vllm serve Qwen/Qwen3-Omni-30B-A3B-Thinking \
      --host 0.0.0.0 \
      --port 8001 \
      --dtype bfloat16 \
      --max-model-len 32768 \
      --gpu-memory-utilization 0.85 \
      --trust-remote-code

[ywang96]

@VivekMalipatel The model works on that particular PR branch of the model support #25550 but not vllm 0.10.2. You can check how to install that branch here https://github.com/QwenLM/Qwen3-Omni?tab=readme-ov-file#vllm-usage

[VivekMalipatel]

This worked for me.

@ywang96 Thank you so much!! I am not sure how did I miss that.

Hope this will soon be supported by the current post-merge releases.