[Bug]: ValueError: Model architectures ['LlamaForCausalLM'] failed to be inspected

[npanpaliya]

Your current environment

Environment:
OS/Arch - IBM linux powerpc64le ubi9 OCP cluster
vLLM version - 0.6.6
torch: 2.5.1
torchvision: 0.20.1

Model Input Dumps

None

🐛 Describe the bug

While deploying a model through kserve inference service that uses vLLM serving runtime, I'm getting below error -

[root@bastion-0 ~]# oc logs tinyllama-predictor-6f7ccc8d86-dfj5r -c kserve-container
INFO 01-03 09:38:16 api_server.py:712] vLLM API server version 0.6.6.post2.dev29+g6036d4db.d20250103
INFO 01-03 09:38:16 api_server.py:713] args: Namespace(host=None, port=8080, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='/mnt/models', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='xgrammar', logits_processor_pattern=None, distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['tinyllama'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False)
INFO 01-03 09:38:16 api_server.py:199] Started engine process with PID 17
ERROR 01-03 09:38:23 registry.py:293] Error in inspecting model architecture 'LlamaForCausalLM'
ERROR 01-03 09:38:23 registry.py:293] Traceback (most recent call last):
ERROR 01-03 09:38:23 registry.py:293]   File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/model_executor/models/registry.py", line 488, in _run_in_subprocess
ERROR 01-03 09:38:23 registry.py:293]     returned.check_returncode()
ERROR 01-03 09:38:23 registry.py:293]   File "/opt/conda/lib/python3.11/subprocess.py", line 502, in check_returncode
ERROR 01-03 09:38:23 registry.py:293]     raise CalledProcessError(self.returncode, self.args, self.stdout,
ERROR 01-03 09:38:23 registry.py:293] subprocess.CalledProcessError: Command '['/opt/conda/bin/python3', '-m', 'vllm.model_executor.models.registry']' died with <Signals.SIGILL: 4>.
ERROR 01-03 09:38:23 registry.py:293] 
ERROR 01-03 09:38:23 registry.py:293] The above exception was the direct cause of the following exception:
ERROR 01-03 09:38:23 registry.py:293] 
ERROR 01-03 09:38:23 registry.py:293] Traceback (most recent call last):
ERROR 01-03 09:38:23 registry.py:293]   File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/model_executor/models/registry.py", line 291, in _try_inspect_model_cls
ERROR 01-03 09:38:23 registry.py:293]     return model.inspect_model_cls()
ERROR 01-03 09:38:23 registry.py:293]            ^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-03 09:38:23 registry.py:293]   File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/model_executor/models/registry.py", line 263, in inspect_model_cls
ERROR 01-03 09:38:23 registry.py:293]     return _run_in_subprocess(
ERROR 01-03 09:38:23 registry.py:293]            ^^^^^^^^^^^^^^^^^^^
ERROR 01-03 09:38:23 registry.py:293]   File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/model_executor/models/registry.py", line 491, in _run_in_subprocess
ERROR 01-03 09:38:23 registry.py:293]     raise RuntimeError(f"Error raised in subprocess:\n"
ERROR 01-03 09:38:23 registry.py:293] RuntimeError: Error raised in subprocess:
ERROR 01-03 09:38:23 registry.py:293] <frozen runpy>:128: RuntimeWarning: 'vllm.model_executor.models.registry' found in sys.modules after import of package 'vllm.model_executor.models', but prior to execution of 'vllm.model_executor.models.registry'; this may result in unpredictable behaviour
ERROR 01-03 09:38:23 registry.py:293] 
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/entrypoints/openai/api_server.py", line 783, in <module>
    uvloop.run(run_server(args))
  File "/opt/conda/lib/python3.11/site-packages/uvloop/__init__.py", line 105, in run
    return runner.run(wrapper())
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
  File "/opt/conda/lib/python3.11/site-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
           ^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/entrypoints/openai/api_server.py", line 749, in run_server
    async with build_async_engine_client(args) as engine_client:
  File "/opt/conda/lib/python3.11/contextlib.py", line 204, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/entrypoints/openai/api_server.py", line 118, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/opt/conda/lib/python3.11/contextlib.py", line 204, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/entrypoints/openai/api_server.py", line 210, in build_async_engine_client_from_engine_args
    engine_config = engine_args.create_engine_config()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/engine/arg_utils.py", line 1044, in create_engine_config
    model_config = self.create_model_config()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/engine/arg_utils.py", line 970, in create_model_config
    return ModelConfig(
           ^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/config.py", line 343, in __init__
    self.multimodal_config = self._init_multimodal_config(
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/config.py", line 398, in _init_multimodal_config
    if ModelRegistry.is_multimodal_model(architectures):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/model_executor/models/registry.py", line 427, in is_multimodal_model
    model_cls, _ = self.inspect_model_cls(architectures)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/model_executor/models/registry.py", line 387, in inspect_model_cls
    return self._raise_for_unsupported(architectures)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/model_executor/models/registry.py", line 344, in _raise_for_unsupported
    raise ValueError(
ValueError: Model architectures ['LlamaForCausalLM'] failed to be inspected. Please check the logs for more details.
ERROR 01-03 09:38:29 registry.py:293] Error in inspecting model architecture 'LlamaForCausalLM'
ERROR 01-03 09:38:29 registry.py:293] Traceback (most recent call last):
ERROR 01-03 09:38:29 registry.py:293]   File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/model_executor/models/registry.py", line 488, in _run_in_subprocess
ERROR 01-03 09:38:29 registry.py:293]     returned.check_returncode()
ERROR 01-03 09:38:29 registry.py:293]   File "/opt/conda/lib/python3.11/subprocess.py", line 502, in check_returncode
ERROR 01-03 09:38:29 registry.py:293]     raise CalledProcessError(self.returncode, self.args, self.stdout,
ERROR 01-03 09:38:29 registry.py:293] subprocess.CalledProcessError: Command '['/opt/conda/bin/python3', '-m', 'vllm.model_executor.models.registry']' died with <Signals.SIGILL: 4>.
ERROR 01-03 09:38:29 registry.py:293] 
ERROR 01-03 09:38:29 registry.py:293] The above exception was the direct cause of the following exception:
ERROR 01-03 09:38:29 registry.py:293] 
ERROR 01-03 09:38:29 registry.py:293] Traceback (most recent call last):
ERROR 01-03 09:38:29 registry.py:293]   File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/model_executor/models/registry.py", line 291, in _try_inspect_model_cls
ERROR 01-03 09:38:29 registry.py:293]     return model.inspect_model_cls()
ERROR 01-03 09:38:29 registry.py:293]            ^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-03 09:38:29 registry.py:293]   File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/model_executor/models/registry.py", line 263, in inspect_model_cls
ERROR 01-03 09:38:29 registry.py:293]     return _run_in_subprocess(
ERROR 01-03 09:38:29 registry.py:293]            ^^^^^^^^^^^^^^^^^^^
ERROR 01-03 09:38:29 registry.py:293]   File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/model_executor/models/registry.py", line 491, in _run_in_subprocess
ERROR 01-03 09:38:29 registry.py:293]     raise RuntimeError(f"Error raised in subprocess:\n"
ERROR 01-03 09:38:29 registry.py:293] RuntimeError: Error raised in subprocess:
ERROR 01-03 09:38:29 registry.py:293] <frozen runpy>:128: RuntimeWarning: 'vllm.model_executor.models.registry' found in sys.modules after import of package 'vllm.model_executor.models', but prior to execution of 'vllm.model_executor.models.registry'; this may result in unpredictable behaviour
ERROR 01-03 09:38:29 registry.py:293] 
ERROR 01-03 09:38:29 engine.py:366] Model architectures ['LlamaForCausalLM'] failed to be inspected. Please check the logs for more details.
ERROR 01-03 09:38:29 engine.py:366] Traceback (most recent call last):
ERROR 01-03 09:38:29 engine.py:366]   File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/engine/multiprocessing/engine.py", line 357, in run_mp_engine
ERROR 01-03 09:38:29 engine.py:366]     engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
ERROR 01-03 09:38:29 engine.py:366]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-03 09:38:29 engine.py:366]   File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/engine/multiprocessing/engine.py", line 114, in from_engine_args
ERROR 01-03 09:38:29 engine.py:366]     engine_config = engine_args.create_engine_config(usage_context)
ERROR 01-03 09:38:29 engine.py:366]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-03 09:38:29 engine.py:366]   File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/engine/arg_utils.py", line 1044, in create_engine_config
ERROR 01-03 09:38:29 engine.py:366]     model_config = self.create_model_config()
ERROR 01-03 09:38:29 engine.py:366]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-03 09:38:29 engine.py:366]   File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/engine/arg_utils.py", line 970, in create_model_config
ERROR 01-03 09:38:29 engine.py:366]     return ModelConfig(
ERROR 01-03 09:38:29 engine.py:366]            ^^^^^^^^^^^^
ERROR 01-03 09:38:29 engine.py:366]   File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/config.py", line 343, in __init__
ERROR 01-03 09:38:29 engine.py:366]     self.multimodal_config = self._init_multimodal_config(
ERROR 01-03 09:38:29 engine.py:366]                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-03 09:38:29 engine.py:366]   File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/config.py", line 398, in _init_multimodal_config
ERROR 01-03 09:38:29 engine.py:366]     if ModelRegistry.is_multimodal_model(architectures):
ERROR 01-03 09:38:29 engine.py:366]        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-03 09:38:29 engine.py:366]   File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/model_executor/models/registry.py", line 427, in is_multimodal_model
ERROR 01-03 09:38:29 engine.py:366]     model_cls, _ = self.inspect_model_cls(architectures)
ERROR 01-03 09:38:29 engine.py:366]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-03 09:38:29 engine.py:366]   File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/model_executor/models/registry.py", line 387, in inspect_model_cls
ERROR 01-03 09:38:29 engine.py:366]     return self._raise_for_unsupported(architectures)
ERROR 01-03 09:38:29 engine.py:366]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-03 09:38:29 engine.py:366]   File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/model_executor/models/registry.py", line 344, in _raise_for_unsupported
ERROR 01-03 09:38:29 engine.py:366]     raise ValueError(
ERROR 01-03 09:38:29 engine.py:366] ValueError: Model architectures ['LlamaForCausalLM'] failed to be inspected. Please check the logs for more details.
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/engine/multiprocessing/engine.py", line 368, in run_mp_engine
    raise e
  File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/engine/multiprocessing/engine.py", line 357, in run_mp_engine
    engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/engine/multiprocessing/engine.py", line 114, in from_engine_args
    engine_config = engine_args.create_engine_config(usage_context)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/engine/arg_utils.py", line 1044, in create_engine_config
    model_config = self.create_model_config()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/engine/arg_utils.py", line 970, in create_model_config
    return ModelConfig(
           ^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/config.py", line 343, in __init__
    self.multimodal_config = self._init_multimodal_config(
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/config.py", line 398, in _init_multimodal_config
    if ModelRegistry.is_multimodal_model(architectures):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/model_executor/models/registry.py", line 427, in is_multimodal_model
    model_cls, _ = self.inspect_model_cls(architectures)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/model_executor/models/registry.py", line 387, in inspect_model_cls
    return self._raise_for_unsupported(architectures)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/model_executor/models/registry.py", line 344, in _raise_for_unsupported
    raise ValueError(
ValueError: Model architectures ['LlamaForCausalLM'] failed to be inspected. Please check the logs for more details

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

ERROR 01-03 09:38:29 registry.py:293]   File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/model_executor/models/registry.py", line 488, in _run_in_subprocess
ERROR 01-03 09:38:29 registry.py:293]     returned.check_returncode()
ERROR 01-03 09:38:29 registry.py:293]   File "/opt/conda/lib/python3.11/subprocess.py", line 502, in check_returncode
ERROR 01-03 09:38:29 registry.py:293]     raise CalledProcessError(self.returncode, self.args, self.stdout,
ERROR 01-03 09:38:29 registry.py:293] subprocess.CalledProcessError: Command '['/opt/conda/bin/python3', '-m', 'vllm.model_executor.models.registry']' died with <Signals.SIGILL: 4>.

That's rather odd. Any idea @youkaichao ?

[npanpaliya]

In the same kserve Raw deployment on an OCP cluster, when I tried deploying facebook/opt125m model, I see a different error -

oc logs opt125m-predictor-7f5479dbff-n4z8m -c kserve-container
INFO 01-03 16:26:32 api_server.py:712] vLLM API server version 0.6.6.post2.dev29+g6036d4db.d20250103
INFO 01-03 16:26:32 api_server.py:713] args: Namespace(host=None, port=8080, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='/mnt/models', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='xgrammar', logits_processor_pattern=None, distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['opt125m'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False)
INFO 01-03 16:26:32 api_server.py:199] Started engine process with PID 17
INFO 01-03 16:26:32 config.py:2249] For POWERPC, we cast models to bfloat16 instead of using float16 by default. Float16 is not currently supported for POWERPC.
WARNING 01-03 16:26:32 config.py:2282] Casting torch.float16 to torch.bfloat16.
INFO 01-03 16:26:40 config.py:2249] For POWERPC, we cast models to bfloat16 instead of using float16 by default. Float16 is not currently supported for POWERPC.
WARNING 01-03 16:26:40 config.py:2282] Casting torch.float16 to torch.bfloat16.
INFO 01-03 16:26:41 config.py:516] This model supports multiple tasks: {'reward', 'classify', 'score', 'embed', 'generate'}. Defaulting to 'generate'.
WARNING 01-03 16:26:41 config.py:648] Async output processing is not supported on the current platform type cpu.
WARNING 01-03 16:26:41 cpu.py:56] CUDA graph is not supported on CPU, fallback to the eager mode.
WARNING 01-03 16:26:41 cpu.py:71] Environment variable VLLM_CPU_KVCACHE_SPACE (GB) for CPU backend is not set, using 4 by default.
[root@odh-cluster-777-bastion-0 ~]# oc logs opt125m-predictor-7f5479dbff-n4z8m -c kserve-container
INFO 01-03 16:26:32 api_server.py:712] vLLM API server version 0.6.6.post2.dev29+g6036d4db.d20250103
INFO 01-03 16:26:32 api_server.py:713] args: Namespace(host=None, port=8080, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='/mnt/models', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='xgrammar', logits_processor_pattern=None, distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['opt125m'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False)
INFO 01-03 16:26:32 api_server.py:199] Started engine process with PID 17
INFO 01-03 16:26:32 config.py:2249] For POWERPC, we cast models to bfloat16 instead of using float16 by default. Float16 is not currently supported for POWERPC.
WARNING 01-03 16:26:32 config.py:2282] Casting torch.float16 to torch.bfloat16.
INFO 01-03 16:26:40 config.py:2249] For POWERPC, we cast models to bfloat16 instead of using float16 by default. Float16 is not currently supported for POWERPC.
WARNING 01-03 16:26:40 config.py:2282] Casting torch.float16 to torch.bfloat16.
INFO 01-03 16:26:41 config.py:516] This model supports multiple tasks: {'reward', 'classify', 'score', 'embed', 'generate'}. Defaulting to 'generate'.
WARNING 01-03 16:26:41 config.py:648] Async output processing is not supported on the current platform type cpu.
WARNING 01-03 16:26:41 cpu.py:56] CUDA graph is not supported on CPU, fallback to the eager mode.
WARNING 01-03 16:26:41 cpu.py:71] Environment variable VLLM_CPU_KVCACHE_SPACE (GB) for CPU backend is not set, using 4 by default.
[root@odh-cluster-777-bastion-0 ~]# oc logs opt125m-predictor-7f5479dbff-n4z8m -c kserve-container
INFO 01-03 16:26:32 api_server.py:712] vLLM API server version 0.6.6.post2.dev29+g6036d4db.d20250103
INFO 01-03 16:26:32 api_server.py:713] args: Namespace(host=None, port=8080, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='/mnt/models', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='xgrammar', logits_processor_pattern=None, distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['opt125m'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False)
INFO 01-03 16:26:32 api_server.py:199] Started engine process with PID 17
INFO 01-03 16:26:32 config.py:2249] For POWERPC, we cast models to bfloat16 instead of using float16 by default. Float16 is not currently supported for POWERPC.
WARNING 01-03 16:26:32 config.py:2282] Casting torch.float16 to torch.bfloat16.
INFO 01-03 16:26:40 config.py:2249] For POWERPC, we cast models to bfloat16 instead of using float16 by default. Float16 is not currently supported for POWERPC.
WARNING 01-03 16:26:40 config.py:2282] Casting torch.float16 to torch.bfloat16.
INFO 01-03 16:26:41 config.py:516] This model supports multiple tasks: {'reward', 'classify', 'score', 'embed', 'generate'}. Defaulting to 'generate'.
WARNING 01-03 16:26:41 config.py:648] Async output processing is not supported on the current platform type cpu.
WARNING 01-03 16:26:41 cpu.py:56] CUDA graph is not supported on CPU, fallback to the eager mode.
WARNING 01-03 16:26:41 cpu.py:71] Environment variable VLLM_CPU_KVCACHE_SPACE (GB) for CPU backend is not set, using 4 by default.
INFO 01-03 16:26:47 config.py:516] This model supports multiple tasks: {'embed', 'generate', 'reward', 'score', 'classify'}. Defaulting to 'generate'.
WARNING 01-03 16:26:47 config.py:648] Async output processing is not supported on the current platform type cpu.
WARNING 01-03 16:26:47 cpu.py:56] CUDA graph is not supported on CPU, fallback to the eager mode.
WARNING 01-03 16:26:47 cpu.py:71] Environment variable VLLM_CPU_KVCACHE_SPACE (GB) for CPU backend is not set, using 4 by default.
INFO 01-03 16:26:47 importing.py:14] Triton not installed or not compatible; certain GPU-related functions will not be available.
INFO 01-03 16:26:47 llm_engine.py:234] Initializing an LLM engine (v0.6.6.post2.dev29+g6036d4db.d20250103) with config: model='/mnt/models', speculative_config=None, tokenizer='/mnt/models', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cpu, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=opt125m, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"candidate_compile_sizes":[],"compile_sizes":[],"capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True, 
INFO 01-03 16:26:48 cpu.py:33] Cannot use _Backend.FLASH_ATTN backend on CPU.
INFO 01-03 16:26:48 selector.py:141] Using Torch SDPA backend.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/entrypoints/openai/api_server.py", line 783, in <module>
    uvloop.run(run_server(args))
  File "/opt/conda/lib/python3.11/site-packages/uvloop/__init__.py", line 105, in run
    return runner.run(wrapper())
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
  File "/opt/conda/lib/python3.11/site-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
           ^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/entrypoints/openai/api_server.py", line 749, in run_server
    async with build_async_engine_client(args) as engine_client:
  File "/opt/conda/lib/python3.11/contextlib.py", line 204, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/entrypoints/openai/api_server.py", line 118, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/opt/conda/lib/python3.11/contextlib.py", line 204, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/entrypoints/openai/api_server.py", line 223, in build_async_engine_client_from_engine_args
    raise RuntimeError(
RuntimeError: Engine process failed to start. See stack trace for the root cause.

If I just manually run root@1adb4ffd1855:/workspace# /opt/conda/bin/python3.11 -m vllm.entrypoints.openai.api_server in the vllm container on different VM, it works fine.

[youkaichao]

are you using cpu or gpu version of vllm?

[npanpaliya]

We are using cpu version of vllm

[Hoipang]

Hello, we had the same issue. It seems to be regardless of models, I've tried SmolLM2-135M & Qwen2-0.5B and both have the same error. We are using the vllm cpu image built using the provided Dockerfile.cpu running in kubernetes.

vllm serve Qwen/Qwen2-0.5B-Instruct-GPTQ --port 8001
vllm serve HuggingFaceTB/SmolLM2-135M-Instruct --port 8001

When attempting to import the vllm.model_executor.models.registry manually, I'm encountering a segmentation fault. Interestingly, this behavior is inconsistent across different environments.

Steps to Reproduce

1. Run the following command:

python3 -c "import vllm.model_executor.models.registry"

2. In some environments, this results in a segmentation fault and terminates the program.
   

3. In other environments, the import succeeds without visible errors.
   

In this environment where the import succeeds, the vllm serve works fine without error. However!!! if I used gdb to backtrace the "import vllm.model_executor.models.registry" once in this working environment. Then it won't work anymore as the python program seems to be able to pickup the Segmentation Fault after the backtrace.

However, when I tried to use GDB to backtrace the issue, even though in the other environments where the import succeeds without visible SegFault errors, GDB still shows there is a segmentation fault happening at the numpy FFT module consistently.

apt install gdb
gdb python3
run -c "import vllm.model_executor.models.registry"

• The segmentation fault occurs in a NumPy FFT module: /usr/local/lib/python3.10/dist-packages/numpy/fft/_pocketfft_internal.cpython-310-x86_64-linux-gnu.so
• The issue is reproducible across multiple attempts
• The behavior is consistent when using GDB, but inconsistent in normal Python execution across different environments
• In the environment where the import succeeds, the vllm serve works fine without error until I used GDB to backtrace the "import vllm.model_executor.models.registry" once ( which shows SegFault ). Then it won't work anymore as the python program seems to be able to pickup the Segmentation Fault after the backtrace. It could also be due to the installation of GDB.

Environment Details

• vLLM version: 0.6.6.post2.dev93+g32c9eff2.d20250106
• Python version: 3.10.12
• NumPy version: 1.26.4

(gdb) run -c "import vllm.model_executor.models.registry"
Starting program: /usr/bin/python3 -c "import vllm.model_executor.models.registry"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Detaching after vfork from child process 91172]
[Detaching after vfork from child process 91173]

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff745ee10 in radfg () from /usr/local/lib/python3.10/dist-packages/numpy/fft/_pocketfft_internal.cpython-310-x86_64-linux-gnu.so
(gdb) bt
#0  0x00007ffff745ee10 in radfg () from /usr/local/lib/python3.10/dist-packages/numpy/fft/_pocketfft_internal.cpython-310-x86_64-linux-gnu.so
#1  0x00005555566c2f00 in ?? ()
#2  0x00005555566c2a00 in ?? ()
#3  0x00005555566c3400 in ?? ()
#4  0x0000555556651700 in ?? ()
#5  0x00005555566c3900 in ?? ()
#6  0x000055555b8fb900 in ?? ()
#7  0x000055555b8fbe00 in ?? ()
#8  0x000055555b8fc300 in ?? ()
#9  0x000055555b8fc800 in ?? ()
#10 0x000055555bafed00 in ?? ()
#11 0x000055555baff700 in ?? ()
#12 0x000055555adf9600 in ?? ()
#13 0x000055555b965700 in ?? ()
#14 0x000055555aefe500 in ?? ()
#15 0x000055555aefea00 in ?? ()
#16 0x000055555af00d00 in ?? ()
#17 0x000055555af01700 in ?? ()
#18 0x000055555af01200 in ?? ()
#19 0x000055555d05c500 in ?? ()
#20 0x000055555d05ca00 in ?? ()
#21 0x000055555d05cf00 in ?? ()
#22 0x000055555d05c000 in ?? ()
#23 0x000055555d05d400 in ?? ()
#24 0x000055555d05d900 in ?? ()
#25 0x000055555d05de00 in ?? ()
#26 0x00007ffff7fbc5f0 in ?? ()
#27 0x000055555d05e300 in ?? ()
#28 0x0000555558f91200 in ?? ()
#29 0x000055555cbc8000 in ?? ()
#30 0x000055555ae90680 in ?? ()
#31 0x000055555cbc8500 in ?? ()
#32 0x000055555cbc9400 in ?? ()
#33 0x000055555cbc9900 in ?? ()
#34 0x000055555d034000 in ?? ()
#35 0x000055555cbc9e00 in ?? ()
#36 0x00007ffff7ca3000 in ?? ()
#37 0x000055555cbca300 in ?? ()
#38 0x000055555cbcb700 in ?? ()
#39 0x000055555d034580 in ?? ()
#40 0x000055555cf50000 in ?? ()
#41 0x000055555d035080 in ?? ()
#42 0x000055555d035600 in ?? ()
#43 0x000055555d035b80 in ?? ()
#44 0x000055555d036100 in ?? ()
#45 0x000055555d036680 in ?? ()
#46 0x000055555d037700 in ?? ()
#47 0x000055555d769200 in ?? ()
#48 0x00007ffff7ca3fc0 in ?? ()
#49 0x00007ffff7fbbbd0 in ?? ()
#50 0x00007ffff7ca44e0 in ?? ()
#51 0x00007ffff7ca3aa0 in ?? ()
#52 0x00007ffff7ca3520 in ?? ()
#53 0x00007ffff7fbcaf0 in ?? ()
#54 0x00007ffff7ffdaf0 in _rtld_global () from /lib64/ld-linux-x86-64.so.2
#55 0x00007ffff7028e80 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#56 0x00007fffffffa9a0 in ?? ()
#57 0x00007fffffffa9a0 in ?? ()
#58 0x00000000f6f3e9f6 in ?? ()
#59 0x00007ffff79c1380 in __frame_dummy_init_array_entry () from /usr/local/lib/libiomp5.so
#60 0x00007ffff79c1388 in __do_global_dtors_aux_fini_array_entry () from /usr/local/lib/libiomp5.so
#61 0x000055555635c9c0 in ?? ()
#62 0x0000000000000000 in ?? ()

[Hoipang]

New Observation.
When I am using docker to run vllm serve, it works fine.
However, if I have installed GDB using "apt install gdb" prior to running vllm serve, then it will fail to run.

Docker with apt install GDB

docker run -it --rm --network=host vllm-cpu-env
root@sake-hp:/workspace/vllm# apt install gdb
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libbabeltrace1 libboost-regex1.74.0 libc6-dbg libdebuginfod-common libdebuginfod1 libdw1 libipt2 libsource-highlight-common libsource-highlight4v5
Suggested packages:
  gdb-doc gdbserver
The following NEW packages will be installed:
  gdb libbabeltrace1 libboost-regex1.74.0 libc6-dbg libdebuginfod-common libdebuginfod1 libdw1 libipt2 libsource-highlight-common libsource-highlight4v5
0 upgraded, 10 newly installed, 0 to remove and 0 not upgraded.
Need to get 19.0 MB of archives.
After this operation, 35.8 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 libdebuginfod-common all 0.186-1build1 [7878 B]
Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 libdw1 amd64 0.186-1build1 [250 kB]
Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 libbabeltrace1 amd64 1.5.8-2build1 [160 kB]
Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 libdebuginfod1 amd64 0.186-1build1 [12.7 kB]
Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 libipt2 amd64 2.0.5-1 [46.4 kB]
Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 libsource-highlight-common all 3.1.9-4.1build2 [64.5 kB]
Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libboost-regex1.74.0 amd64 1.74.0-14ubuntu3 [511 kB]
Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libsource-highlight4v5 amd64 3.1.9-4.1build2 [207 kB]
Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gdb amd64 12.1-0ubuntu1~22.04.2 [3920 kB]
Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libc6-dbg amd64 2.35-0ubuntu3.8 [13.8 MB]
Fetched 19.0 MB in 3s (6774 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libdebuginfod-common.
(Reading database ... 23040 files and directories currently installed.)
Preparing to unpack .../0-libdebuginfod-common_0.186-1build1_all.deb ...
Unpacking libdebuginfod-common (0.186-1build1) ...
Selecting previously unselected package libdw1:amd64.
Preparing to unpack .../1-libdw1_0.186-1build1_amd64.deb ...
Unpacking libdw1:amd64 (0.186-1build1) ...
Selecting previously unselected package libbabeltrace1:amd64.
Preparing to unpack .../2-libbabeltrace1_1.5.8-2build1_amd64.deb ...
Unpacking libbabeltrace1:amd64 (1.5.8-2build1) ...
Selecting previously unselected package libdebuginfod1:amd64.
Preparing to unpack .../3-libdebuginfod1_0.186-1build1_amd64.deb ...
Unpacking libdebuginfod1:amd64 (0.186-1build1) ...
Selecting previously unselected package libipt2.
Preparing to unpack .../4-libipt2_2.0.5-1_amd64.deb ...
Unpacking libipt2 (2.0.5-1) ...
Selecting previously unselected package libsource-highlight-common.
Preparing to unpack .../5-libsource-highlight-common_3.1.9-4.1build2_all.deb ...
Unpacking libsource-highlight-common (3.1.9-4.1build2) ...
Selecting previously unselected package libboost-regex1.74.0:amd64.
Preparing to unpack .../6-libboost-regex1.74.0_1.74.0-14ubuntu3_amd64.deb ...
Unpacking libboost-regex1.74.0:amd64 (1.74.0-14ubuntu3) ...
Selecting previously unselected package libsource-highlight4v5.
Preparing to unpack .../7-libsource-highlight4v5_3.1.9-4.1build2_amd64.deb ...
Unpacking libsource-highlight4v5 (3.1.9-4.1build2) ...
Selecting previously unselected package gdb.
Preparing to unpack .../8-gdb_12.1-0ubuntu1~22.04.2_amd64.deb ...
Unpacking gdb (12.1-0ubuntu1~22.04.2) ...
Selecting previously unselected package libc6-dbg:amd64.
Preparing to unpack .../9-libc6-dbg_2.35-0ubuntu3.8_amd64.deb ...
Unpacking libc6-dbg:amd64 (2.35-0ubuntu3.8) ...
Setting up libdw1:amd64 (0.186-1build1) ...
Setting up libdebuginfod-common (0.186-1build1) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 78.)
debconf: falling back to frontend: Readline

Creating config file /etc/profile.d/debuginfod.sh with new version

Creating config file /etc/profile.d/debuginfod.csh with new version
Setting up libdebuginfod1:amd64 (0.186-1build1) ...
Setting up libsource-highlight-common (3.1.9-4.1build2) ...
Setting up libc6-dbg:amd64 (2.35-0ubuntu3.8) ...
Setting up libboost-regex1.74.0:amd64 (1.74.0-14ubuntu3) ...
Setting up libipt2 (2.0.5-1) ...
Setting up libbabeltrace1:amd64 (1.5.8-2build1) ...
Setting up libsource-highlight4v5 (3.1.9-4.1build2) ...
Setting up gdb (12.1-0ubuntu1~22.04.2) ...
Processing triggers for libc-bin (2.35-0ubuntu3.8) ...
/sbin/ldconfig.real: /usr/local/lib/libur_loader.so.0 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libhwloc.so.15 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libumf.so.0 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libur_adapter_opencl.so.0 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtcm_debug.so.1 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtcm.so.1 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libur_adapter_level_zero.so.0 is not a symbolic link

root@sake-hp:/workspace/vllm# apt install gdb^C
root@sake-hp:/workspace/vllm# vllm serve HuggingFaceTB/SmolLM2-135M-Instruct --port 8001
INFO 01-07 09:57:27 api_server.py:764] vLLM API server version 0.6.6.post2.dev93+g32c9eff2.d20250106
INFO 01-07 09:57:27 api_server.py:765] args: Namespace(subparser='serve', model_tag='HuggingFaceTB/SmolLM2-135M-Instruct', config='', host=None, port=8001, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='HuggingFaceTB/SmolLM2-135M-Instruct', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='xgrammar', logits_processor_pattern=None, distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, dispatch_function=<function serve at 0x743cc6f37130>)
INFO 01-07 09:57:28 __init__.py:179] Automatically detected platform cpu.
INFO 01-07 09:57:28 api_server.py:194] Started engine process with PID 288
config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 861/861 [00:00<00:00, 5.50MB/s]
INFO 01-07 09:57:34 __init__.py:179] Automatically detected platform cpu.
ERROR 01-07 09:57:39 registry.py:294] Error in inspecting model architecture 'LlamaForCausalLM'
ERROR 01-07 09:57:39 registry.py:294] Traceback (most recent call last):
ERROR 01-07 09:57:39 registry.py:294]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/registry.py", line 489, in _run_in_subprocess
ERROR 01-07 09:57:39 registry.py:294]     returned.check_returncode()
ERROR 01-07 09:57:39 registry.py:294]   File "/usr/lib/python3.10/subprocess.py", line 457, in check_returncode
ERROR 01-07 09:57:39 registry.py:294]     raise CalledProcessError(self.returncode, self.args, self.stdout,
ERROR 01-07 09:57:39 registry.py:294] subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'vllm.model_executor.models.registry']' died with <Signals.SIGSEGV: 11>.
ERROR 01-07 09:57:39 registry.py:294]
ERROR 01-07 09:57:39 registry.py:294] The above exception was the direct cause of the following exception:
ERROR 01-07 09:57:39 registry.py:294]
ERROR 01-07 09:57:39 registry.py:294] Traceback (most recent call last):
ERROR 01-07 09:57:39 registry.py:294]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/registry.py", line 292, in _try_inspect_model_cls
ERROR 01-07 09:57:39 registry.py:294]     return model.inspect_model_cls()
ERROR 01-07 09:57:39 registry.py:294]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/registry.py", line 263, in inspect_model_cls
ERROR 01-07 09:57:39 registry.py:294]     return _run_in_subprocess(
ERROR 01-07 09:57:39 registry.py:294]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/registry.py", line 492, in _run_in_subprocess
ERROR 01-07 09:57:39 registry.py:294]     raise RuntimeError(f"Error raised in subprocess:\n"
ERROR 01-07 09:57:39 registry.py:294] RuntimeError: Error raised in subprocess:
ERROR 01-07 09:57:39 registry.py:294] /usr/lib/python3.10/runpy.py:126: RuntimeWarning: 'vllm.model_executor.models.registry' found in sys.modules after import of package 'vllm.model_executor.models', but prior to execution of 'vllm.model_executor.models.registry'; this may result in unpredictable behaviour
ERROR 01-07 09:57:39 registry.py:294]   warn(RuntimeWarning(msg))
ERROR 01-07 09:57:39 registry.py:294]
Traceback (most recent call last):
  File "/usr/local/bin/vllm", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/vllm/scripts.py", line 201, in main
    args.dispatch_function(args)
  File "/usr/local/lib/python3.10/dist-packages/vllm/scripts.py", line 42, in serve
    uvloop.run(run_server(args))
  File "/usr/local/lib/python3.10/dist-packages/uvloop/__init__.py", line 82, in run
    return loop.run_until_complete(wrapper())
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/usr/local/lib/python3.10/dist-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 792, in run_server
    async with build_async_engine_client(args) as engine_client:
  File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 124, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 205, in build_async_engine_client_from_engine_args
    engine_config = engine_args.create_engine_config()
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/arg_utils.py", line 1043, in create_engine_config
    model_config = self.create_model_config()
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/arg_utils.py", line 969, in create_model_config
    return ModelConfig(
  File "/usr/local/lib/python3.10/dist-packages/vllm/config.py", line 343, in __init__
    self.multimodal_config = self._init_multimodal_config(
  File "/usr/local/lib/python3.10/dist-packages/vllm/config.py", line 399, in _init_multimodal_config
    if ModelRegistry.is_multimodal_model(architectures):
  File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/registry.py", line 428, in is_multimodal_model
    model_cls, _ = self.inspect_model_cls(architectures)
  File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/registry.py", line 388, in inspect_model_cls
    return self._raise_for_unsupported(architectures)
  File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/registry.py", line 345, in _raise_for_unsupported
    raise ValueError(
ValueError: Model architectures ['LlamaForCausalLM'] failed to be inspected. Please check the logs for more details.
ERROR 01-07 09:57:44 registry.py:294] Error in inspecting model architecture 'LlamaForCausalLM'
ERROR 01-07 09:57:44 registry.py:294] Traceback (most recent call last):
ERROR 01-07 09:57:44 registry.py:294]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/registry.py", line 489, in _run_in_subprocess
ERROR 01-07 09:57:44 registry.py:294]     returned.check_returncode()
ERROR 01-07 09:57:44 registry.py:294]   File "/usr/lib/python3.10/subprocess.py", line 457, in check_returncode
ERROR 01-07 09:57:44 registry.py:294]     raise CalledProcessError(self.returncode, self.args, self.stdout,
ERROR 01-07 09:57:44 registry.py:294] subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'vllm.model_executor.models.registry']' died with <Signals.SIGSEGV: 11>.
ERROR 01-07 09:57:44 registry.py:294]
ERROR 01-07 09:57:44 registry.py:294] The above exception was the direct cause of the following exception:
ERROR 01-07 09:57:44 registry.py:294]
ERROR 01-07 09:57:44 registry.py:294] Traceback (most recent call last):
ERROR 01-07 09:57:44 registry.py:294]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/registry.py", line 292, in _try_inspect_model_cls
ERROR 01-07 09:57:44 registry.py:294]     return model.inspect_model_cls()
ERROR 01-07 09:57:44 registry.py:294]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/registry.py", line 263, in inspect_model_cls
ERROR 01-07 09:57:44 registry.py:294]     return _run_in_subprocess(
ERROR 01-07 09:57:44 registry.py:294]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/registry.py", line 492, in _run_in_subprocess
ERROR 01-07 09:57:44 registry.py:294]     raise RuntimeError(f"Error raised in subprocess:\n"
ERROR 01-07 09:57:44 registry.py:294] RuntimeError: Error raised in subprocess:
ERROR 01-07 09:57:44 registry.py:294] /usr/lib/python3.10/runpy.py:126: RuntimeWarning: 'vllm.model_executor.models.registry' found in sys.modules after import of package 'vllm.model_executor.models', but prior to execution of 'vllm.model_executor.models.registry'; this may result in unpredictable behaviour
ERROR 01-07 09:57:44 registry.py:294]   warn(RuntimeWarning(msg))
ERROR 01-07 09:57:44 registry.py:294]
ERROR 01-07 09:57:44 engine.py:366] Model architectures ['LlamaForCausalLM'] failed to be inspected. Please check the logs for more details.
ERROR 01-07 09:57:44 engine.py:366] Traceback (most recent call last):
ERROR 01-07 09:57:44 engine.py:366]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/multiprocessing/engine.py", line 357, in run_mp_engine
ERROR 01-07 09:57:44 engine.py:366]     engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
ERROR 01-07 09:57:44 engine.py:366]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/multiprocessing/engine.py", line 114, in from_engine_args
ERROR 01-07 09:57:44 engine.py:366]     engine_config = engine_args.create_engine_config(usage_context)
ERROR 01-07 09:57:44 engine.py:366]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/arg_utils.py", line 1043, in create_engine_config
ERROR 01-07 09:57:44 engine.py:366]     model_config = self.create_model_config()
ERROR 01-07 09:57:44 engine.py:366]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/arg_utils.py", line 969, in create_model_config
ERROR 01-07 09:57:44 engine.py:366]     return ModelConfig(
ERROR 01-07 09:57:44 engine.py:366]   File "/usr/local/lib/python3.10/dist-packages/vllm/config.py", line 343, in __init__
ERROR 01-07 09:57:44 engine.py:366]     self.multimodal_config = self._init_multimodal_config(
ERROR 01-07 09:57:44 engine.py:366]   File "/usr/local/lib/python3.10/dist-packages/vllm/config.py", line 399, in _init_multimodal_config
ERROR 01-07 09:57:44 engine.py:366]     if ModelRegistry.is_multimodal_model(architectures):
ERROR 01-07 09:57:44 engine.py:366]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/registry.py", line 428, in is_multimodal_model
ERROR 01-07 09:57:44 engine.py:366]     model_cls, _ = self.inspect_model_cls(architectures)
ERROR 01-07 09:57:44 engine.py:366]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/registry.py", line 388, in inspect_model_cls
ERROR 01-07 09:57:44 engine.py:366]     return self._raise_for_unsupported(architectures)
ERROR 01-07 09:57:44 engine.py:366]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/registry.py", line 345, in _raise_for_unsupported
ERROR 01-07 09:57:44 engine.py:366]     raise ValueError(
ERROR 01-07 09:57:44 engine.py:366] ValueError: Model architectures ['LlamaForCausalLM'] failed to be inspected. Please check the logs for more details.
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/multiprocessing/engine.py", line 368, in run_mp_engine
    raise e
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/multiprocessing/engine.py", line 357, in run_mp_engine
    engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/multiprocessing/engine.py", line 114, in from_engine_args
    engine_config = engine_args.create_engine_config(usage_context)
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/arg_utils.py", line 1043, in create_engine_config
    model_config = self.create_model_config()
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/arg_utils.py", line 969, in create_model_config
    return ModelConfig(
  File "/usr/local/lib/python3.10/dist-packages/vllm/config.py", line 343, in __init__
    self.multimodal_config = self._init_multimodal_config(
  File "/usr/local/lib/python3.10/dist-packages/vllm/config.py", line 399, in _init_multimodal_config
    if ModelRegistry.is_multimodal_model(architectures):
  File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/registry.py", line 428, in is_multimodal_model
    model_cls, _ = self.inspect_model_cls(architectures)
  File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/registry.py", line 388, in inspect_model_cls
    return self._raise_for_unsupported(architectures)
  File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/registry.py", line 345, in _raise_for_unsupported
    raise ValueError(
ValueError: Model architectures ['LlamaForCausalLM'] failed to be inspected. Please check the logs for more details.
Segmentation fault (core dumped)

Docker without apt install gdb

docker run -it --rm --network=host vllm-cpu-env
root@sake-hp:/workspace/vllm# vllm serve HuggingFaceTB/SmolLM2-135M-Instruct --port 8001
INFO 01-07 09:59:07 api_server.py:764] vLLM API server version 0.6.6.post2.dev93+g32c9eff2.d20250106
INFO 01-07 09:59:07 api_server.py:765] args: Namespace(subparser='serve', model_tag='HuggingFaceTB/SmolLM2-135M-Instruct', config='', host=None, port=8001, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='HuggingFaceTB/SmolLM2-135M-Instruct', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='xgrammar', logits_processor_pattern=None, distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, dispatch_function=<function serve at 0x743454ed3130>)
INFO 01-07 09:59:08 __init__.py:179] Automatically detected platform cpu.
INFO 01-07 09:59:08 api_server.py:194] Started engine process with PID 22
config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 861/861 [00:00<00:00, 5.85MB/s]
INFO 01-07 09:59:14 __init__.py:179] Automatically detected platform cpu.
INFO 01-07 09:59:21 config.py:517] This model supports multiple tasks: {'reward', 'classify', 'generate', 'embed', 'score'}. Defaulting to 'generate'.
WARNING 01-07 09:59:21 config.py:651] Async output processing is not supported on the current platform type cpu.
WARNING 01-07 09:59:21 _logger.py:72] CUDA graph is not supported on CPU, fallback to the eager mode.
WARNING 01-07 09:59:21 _logger.py:72] Environment variable VLLM_CPU_KVCACHE_SPACE (GB) for CPU backend is not set, using 4 by default.
tokenizer_config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 3.76k/3.76k [00:00<00:00, 20.3MB/s]
vocab.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 801k/801k [00:00<00:00, 1.19MB/s]
merges.txt:   0%|                                                                                                                | 0.00/466k [00:00<?, ?B/s]INFO 01-07 09:59:24 config.py:517] This model supports multiple tasks: {'embed', 'reward', 'generate', 'classify', 'score'}. Defaulting to 'generate'.
WARNING 01-07 09:59:24 config.py:651] Async output processing is not supported on the current platform type cpu.
WARNING 01-07 09:59:24 _logger.py:72] CUDA graph is not supported on CPU, fallback to the eager mode.
WARNING 01-07 09:59:24 _logger.py:72] Environment variable VLLM_CPU_KVCACHE_SPACE (GB) for CPU backend is not set, using 4 by default.
INFO 01-07 09:59:24 importing.py:14] Triton not installed or not compatible; certain GPU-related functions will not be available.
INFO 01-07 09:59:24 llm_engine.py:234] Initializing an LLM engine (v0.6.6.post2.dev93+g32c9eff2.d20250106) with config: model='HuggingFaceTB/SmolLM2-135M-Instruct', speculative_config=None, tokenizer='HuggingFaceTB/SmolLM2-135M-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cpu, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=HuggingFaceTB/SmolLM2-135M-Instruct, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"candidate_compile_sizes":[],"compile_sizes":[],"capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True,
merges.txt: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 466k/466k [00:00<00:00, 1.04MB/s]
tokenizer.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 2.10M/2.10M [00:00<00:00, 4.68MB/s]
special_tokens_map.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 655/655 [00:00<00:00, 3.59MB/s]
generation_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 132/132 [00:00<00:00, 1.14MB/s]
INFO 01-07 09:59:27 cpu.py:33] Cannot use _Backend.FLASH_ATTN backend on CPU.
INFO 01-07 09:59:27 selector.py:141] Using Torch SDPA backend.
INFO 01-07 09:59:27 weight_utils.py:251] Using model weights format ['*.safetensors']
model.safetensors: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 269M/269M [00:06<00:00, 42.4MB/s]
INFO 01-07 09:59:35 weight_utils.py:296] No model.safetensors.index.json found in remote.
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  8.23it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  8.20it/s]

INFO 01-07 09:59:35 cpu_executor.py:186] # CPU blocks: 11650
INFO 01-07 09:59:37 llm_engine.py:431] init engine (profile, create kv cache, warmup model) took 2.03 seconds
INFO 01-07 09:59:38 api_server.py:689] Using supplied chat template:
INFO 01-07 09:59:38 api_server.py:689] None
INFO 01-07 09:59:38 launcher.py:19] Available routes are:
INFO 01-07 09:59:38 launcher.py:27] Route: /openapi.json, Methods: GET, HEAD
INFO 01-07 09:59:38 launcher.py:27] Route: /docs, Methods: GET, HEAD
INFO 01-07 09:59:38 launcher.py:27] Route: /docs/oauth2-redirect, Methods: GET, HEAD
INFO 01-07 09:59:38 launcher.py:27] Route: /redoc, Methods: GET, HEAD
INFO 01-07 09:59:38 launcher.py:27] Route: /health, Methods: GET
INFO 01-07 09:59:38 launcher.py:27] Route: /ping, Methods: POST, GET
INFO 01-07 09:59:38 launcher.py:27] Route: /tokenize, Methods: POST
INFO 01-07 09:59:38 launcher.py:27] Route: /detokenize, Methods: POST
INFO 01-07 09:59:38 launcher.py:27] Route: /v1/models, Methods: GET
INFO 01-07 09:59:38 launcher.py:27] Route: /version, Methods: GET
INFO 01-07 09:59:38 launcher.py:27] Route: /v1/chat/completions, Methods: POST
INFO 01-07 09:59:38 launcher.py:27] Route: /v1/completions, Methods: POST
INFO 01-07 09:59:38 launcher.py:27] Route: /v1/embeddings, Methods: POST
INFO 01-07 09:59:38 launcher.py:27] Route: /pooling, Methods: POST
INFO 01-07 09:59:38 launcher.py:27] Route: /score, Methods: POST
INFO 01-07 09:59:38 launcher.py:27] Route: /v1/score, Methods: POST
INFO 01-07 09:59:38 launcher.py:27] Route: /invocations, Methods: POST
INFO:     Started server process [13]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)

[npanpaliya]

Yeah, right. For me as well, simple docker run has worked. Haven't tried with gdb. But this is strange, gdb should not affect this flow.

root@1adb4ffd1855:/workspace# /opt/conda/bin/python3.11 -m vllm.entrypoints.openai.api_server
INFO 01-06 09:40:28 api_server.py:712] vLLM API server version 0.6.6.post2.dev29+g6036d4db.d20250103
INFO 01-06 09:40:28 api_server.py:714] args: Namespace(host=None, port=8005, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='facebook/opt-125m', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='xgrammar', logits_processor_pattern=None, distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False)
INFO 01-06 09:40:28 api_server.py:199] Started engine process with PID 977
INFO 01-06 09:40:38 config.py:2249] For POWERPC, we cast models to bfloat16 instead of using float16 by default. Float16 is not currently supported for POWERPC.
WARNING 01-06 09:40:38 config.py:2282] Casting torch.float16 to torch.bfloat16.
INFO 01-06 09:40:42 config.py:2249] For POWERPC, we cast models to bfloat16 instead of using float16 by default. Float16 is not currently supported for POWERPC.
WARNING 01-06 09:40:42 config.py:2282] Casting torch.float16 to torch.bfloat16.
INFO 01-06 09:40:44 config.py:516] This model supports multiple tasks: {'reward', 'embed', 'classify', 'generate', 'score'}. Defaulting to 'generate'.
WARNING 01-06 09:40:44 config.py:648] Async output processing is not supported on the current platform type cpu.
WARNING 01-06 09:40:44 cpu.py:56] CUDA graph is not supported on CPU, fallback to the eager mode.
WARNING 01-06 09:40:44 cpu.py:71] Environment variable VLLM_CPU_KVCACHE_SPACE (GB) for CPU backend is not set, using 4 by default.
INFO 01-06 09:40:47 config.py:516] This model supports multiple tasks: {'generate', 'score', 'embed', 'classify', 'reward'}. Defaulting to 'generate'.
WARNING 01-06 09:40:47 config.py:648] Async output processing is not supported on the current platform type cpu.
WARNING 01-06 09:40:47 cpu.py:56] CUDA graph is not supported on CPU, fallback to the eager mode.
WARNING 01-06 09:40:47 cpu.py:71] Environment variable VLLM_CPU_KVCACHE_SPACE (GB) for CPU backend is not set, using 4 by default.
INFO 01-06 09:40:47 importing.py:14] Triton not installed or not compatible; certain GPU-related functions will not be available.
INFO 01-06 09:40:47 llm_engine.py:234] Initializing an LLM engine (v0.6.6.post2.dev29+g6036d4db.d20250103) with config: model='facebook/opt-125m', speculative_config=None, tokenizer='facebook/opt-125m', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cpu, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=facebook/opt-125m, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"candidate_compile_sizes":[],"compile_sizes":[],"capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True, 
INFO 01-06 09:40:49 cpu.py:33] Cannot use _Backend.FLASH_ATTN backend on CPU.
INFO 01-06 09:40:49 selector.py:141] Using Torch SDPA backend.
INFO 01-06 09:40:49 weight_utils.py:251] Using model weights format ['*.bin']
Loading pt checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/model_executor/model_loader/weight_utils.py:450: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  state = torch.load(bin_file, map_location="cpu")
Loading pt checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  6.25it/s]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  6.23it/s]

INFO 01-06 09:40:50 cpu_executor.py:186] # CPU blocks: 7281
INFO 01-06 09:40:50 llm_engine.py:431] init engine (profile, create kv cache, warmup model) took 0.07 seconds
INFO 01-06 09:40:52 api_server.py:640] Using supplied chat template:
INFO 01-06 09:40:52 api_server.py:640] None
INFO 01-06 09:40:52 launcher.py:19] Available routes are:
INFO 01-06 09:40:52 launcher.py:27] Route: /openapi.json, Methods: HEAD, GET
INFO 01-06 09:40:52 launcher.py:27] Route: /docs, Methods: HEAD, GET
INFO 01-06 09:40:52 launcher.py:27] Route: /docs/oauth2-redirect, Methods: HEAD, GET
INFO 01-06 09:40:52 launcher.py:27] Route: /redoc, Methods: HEAD, GET
INFO 01-06 09:40:52 launcher.py:27] Route: /health, Methods: GET
INFO 01-06 09:40:52 launcher.py:27] Route: /tokenize, Methods: POST
INFO 01-06 09:40:52 launcher.py:27] Route: /detokenize, Methods: POST
INFO 01-06 09:40:52 launcher.py:27] Route: /v1/models, Methods: GET
INFO 01-06 09:40:52 launcher.py:27] Route: /version, Methods: GET
INFO 01-06 09:40:52 launcher.py:27] Route: /v1/chat/completions, Methods: POST
INFO 01-06 09:40:52 launcher.py:27] Route: /v1/completions, Methods: POST
INFO 01-06 09:40:52 launcher.py:27] Route: /v1/embeddings, Methods: POST
INFO 01-06 09:40:52 launcher.py:27] Route: /pooling, Methods: POST
INFO 01-06 09:40:52 launcher.py:27] Route: /score, Methods: POST
INFO 01-06 09:40:52 launcher.py:27] Route: /v1/score, Methods: POST
INFO:     Started server process [942]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8005 (Press CTRL+C to quit)

[npanpaliya]

One more observation - I see the same error even with older vllm versions like 0.5, 0.6.1, 0.6.4, which used to work earlier.

[npanpaliya]

Just tried by installing GDB too. It still worked in a standalone docker container.

root@1adb4ffd1855:/workspace# apt-get install gdb
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libbabeltrace1 libboost-regex1.74.0 libc6-dbg libdebuginfod-common libdebuginfod1 libdw1 libpython3.11 libsource-highlight-common libsource-highlight4v5
  sensible-utils ucf
Suggested packages:
  gdb-doc gdbserver
The following NEW packages will be installed:
  gdb libbabeltrace1 libboost-regex1.74.0 libc6-dbg libdebuginfod-common libdebuginfod1 libdw1 libpython3.11 libsource-highlight-common libsource-highlight4v5
  sensible-utils ucf
0 upgraded, 12 newly installed, 0 to remove and 0 not upgraded.
Need to get 15.5 MB of archives.
After this operation, 42.3 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://deb.debian.org/debian bookworm/main ppc64el sensible-utils all 0.0.17+nmu1 [19.0 kB]
Get:2 http://deb.debian.org/debian bookworm/main ppc64el ucf all 3.0043+nmu1 [55.2 kB]
Get:3 http://deb.debian.org/debian bookworm/main ppc64el libdebuginfod-common all 0.188-2.1 [21.5 kB]
Get:4 http://deb.debian.org/debian bookworm/main ppc64el libdw1 ppc64el 0.188-2.1 [253 kB]
Get:5 http://deb.debian.org/debian bookworm/main ppc64el libbabeltrace1 ppc64el 1.5.11-1+b2 [189 kB]
Get:6 http://deb.debian.org/debian bookworm/main ppc64el libdebuginfod1 ppc64el 0.188-2.1 [28.7 kB]
Get:7 http://deb.debian.org/debian bookworm/main ppc64el libpython3.11 ppc64el 3.11.2-6+deb12u4 [2085 kB]
Get:8 http://deb.debian.org/debian bookworm/main ppc64el libsource-highlight-common all 3.1.9-4.2 [77.4 kB]
Get:9 http://deb.debian.org/debian bookworm/main ppc64el libboost-regex1.74.0 ppc64el 1.74.0+ds1-21 [487 kB]
Get:10 http://deb.debian.org/debian bookworm/main ppc64el libsource-highlight4v5 ppc64el 3.1.9-4.2+b3 [262 kB]
Get:11 http://deb.debian.org/debian bookworm/main ppc64el gdb ppc64el 13.1-3 [3926 kB]
Get:12 http://deb.debian.org/debian bookworm/main ppc64el libc6-dbg ppc64el 2.36-9+deb12u9 [8133 kB]
Fetched 15.5 MB in 1s (13.9 MB/s)    
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package sensible-utils.
(Reading database ... 27315 files and directories currently installed.)
Preparing to unpack .../00-sensible-utils_0.0.17+nmu1_all.deb ...
Unpacking sensible-utils (0.0.17+nmu1) ...
Selecting previously unselected package ucf.
Preparing to unpack .../01-ucf_3.0043+nmu1_all.deb ...
Moving old data out of the way
Unpacking ucf (3.0043+nmu1) ...
Selecting previously unselected package libdebuginfod-common.
Preparing to unpack .../02-libdebuginfod-common_0.188-2.1_all.deb ...
Unpacking libdebuginfod-common (0.188-2.1) ...
Selecting previously unselected package libdw1:ppc64el.
Preparing to unpack .../03-libdw1_0.188-2.1_ppc64el.deb ...
Unpacking libdw1:ppc64el (0.188-2.1) ...
Selecting previously unselected package libbabeltrace1:ppc64el.
Preparing to unpack .../04-libbabeltrace1_1.5.11-1+b2_ppc64el.deb ...
Unpacking libbabeltrace1:ppc64el (1.5.11-1+b2) ...
Selecting previously unselected package libdebuginfod1:ppc64el.
Preparing to unpack .../05-libdebuginfod1_0.188-2.1_ppc64el.deb ...
Unpacking libdebuginfod1:ppc64el (0.188-2.1) ...
Selecting previously unselected package libpython3.11:ppc64el.
Preparing to unpack .../06-libpython3.11_3.11.2-6+deb12u4_ppc64el.deb ...
Unpacking libpython3.11:ppc64el (3.11.2-6+deb12u4) ...
Selecting previously unselected package libsource-highlight-common.
Preparing to unpack .../07-libsource-highlight-common_3.1.9-4.2_all.deb ...
Unpacking libsource-highlight-common (3.1.9-4.2) ...
Selecting previously unselected package libboost-regex1.74.0:ppc64el.
Preparing to unpack .../08-libboost-regex1.74.0_1.74.0+ds1-21_ppc64el.deb ...
Unpacking libboost-regex1.74.0:ppc64el (1.74.0+ds1-21) ...
Selecting previously unselected package libsource-highlight4v5:ppc64el.
Preparing to unpack .../09-libsource-highlight4v5_3.1.9-4.2+b3_ppc64el.deb ...
Unpacking libsource-highlight4v5:ppc64el (3.1.9-4.2+b3) ...
Selecting previously unselected package gdb.
Preparing to unpack .../10-gdb_13.1-3_ppc64el.deb ...
Unpacking gdb (13.1-3) ...
Selecting previously unselected package libc6-dbg:ppc64el.
Preparing to unpack .../11-libc6-dbg_2.36-9+deb12u9_ppc64el.deb ...
Unpacking libc6-dbg:ppc64el (2.36-9+deb12u9) ...
Setting up libdw1:ppc64el (0.188-2.1) ...
Setting up libpython3.11:ppc64el (3.11.2-6+deb12u4) ...
Setting up libsource-highlight-common (3.1.9-4.2) ...
Setting up libc6-dbg:ppc64el (2.36-9+deb12u9) ...
Setting up libboost-regex1.74.0:ppc64el (1.74.0+ds1-21) ...
Setting up libbabeltrace1:ppc64el (1.5.11-1+b2) ...
Setting up sensible-utils (0.0.17+nmu1) ...
Setting up libsource-highlight4v5:ppc64el (3.1.9-4.2+b3) ...
Setting up ucf (3.0043+nmu1) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 78.)
debconf: falling back to frontend: Readline
Setting up libdebuginfod-common (0.188-2.1) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 78.)
debconf: falling back to frontend: Readline
Setting up libdebuginfod1:ppc64el (0.188-2.1) ...
Setting up gdb (13.1-3) ...
Processing triggers for libc-bin (2.36-9+deb12u9) ...
root@1adb4ffd1855:/workspace# 
root@1adb4ffd1855:/workspace# 
root@1adb4ffd1855:/workspace# 
root@1adb4ffd1855:/workspace# 
root@1adb4ffd1855:/workspace# 
root@1adb4ffd1855:/workspace# /opt/conda/bin/python3 -m vllm.entrypoints.openai.api_server
INFO 01-07 10:12:37 api_server.py:712] vLLM API server version 0.6.6.post2.dev29+g6036d4db.d20250103
INFO 01-07 10:12:37 api_server.py:714] args: Namespace(host=None, port=8005, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='facebook/opt-125m', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='xgrammar', logits_processor_pattern=None, distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False)
INFO 01-07 10:12:37 api_server.py:199] Started engine process with PID 1587
INFO 01-07 10:12:47 config.py:2249] For POWERPC, we cast models to bfloat16 instead of using float16 by default. Float16 is not currently supported for POWERPC.
WARNING 01-07 10:12:47 config.py:2282] Casting torch.float16 to torch.bfloat16.
INFO 01-07 10:12:52 config.py:2249] For POWERPC, we cast models to bfloat16 instead of using float16 by default. Float16 is not currently supported for POWERPC.
WARNING 01-07 10:12:52 config.py:2282] Casting torch.float16 to torch.bfloat16.
INFO 01-07 10:12:54 config.py:516] This model supports multiple tasks: {'score', 'generate', 'classify', 'embed', 'reward'}. Defaulting to 'generate'.
WARNING 01-07 10:12:54 config.py:648] Async output processing is not supported on the current platform type cpu.
WARNING 01-07 10:12:54 cpu.py:56] CUDA graph is not supported on CPU, fallback to the eager mode.
WARNING 01-07 10:12:54 cpu.py:71] Environment variable VLLM_CPU_KVCACHE_SPACE (GB) for CPU backend is not set, using 4 by default.
INFO 01-07 10:12:58 config.py:516] This model supports multiple tasks: {'score', 'embed', 'classify', 'reward', 'generate'}. Defaulting to 'generate'.
WARNING 01-07 10:12:58 config.py:648] Async output processing is not supported on the current platform type cpu.
WARNING 01-07 10:12:58 cpu.py:56] CUDA graph is not supported on CPU, fallback to the eager mode.
WARNING 01-07 10:12:58 cpu.py:71] Environment variable VLLM_CPU_KVCACHE_SPACE (GB) for CPU backend is not set, using 4 by default.
INFO 01-07 10:12:58 importing.py:14] Triton not installed or not compatible; certain GPU-related functions will not be available.
INFO 01-07 10:12:58 llm_engine.py:234] Initializing an LLM engine (v0.6.6.post2.dev29+g6036d4db.d20250103) with config: model='facebook/opt-125m', speculative_config=None, tokenizer='facebook/opt-125m', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cpu, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=facebook/opt-125m, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"candidate_compile_sizes":[],"compile_sizes":[],"capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True, 
INFO 01-07 10:13:00 cpu.py:33] Cannot use _Backend.FLASH_ATTN backend on CPU.
INFO 01-07 10:13:00 selector.py:141] Using Torch SDPA backend.
INFO 01-07 10:13:00 weight_utils.py:251] Using model weights format ['*.bin']
Loading pt checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/model_executor/model_loader/weight_utils.py:450: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  state = torch.load(bin_file, map_location="cpu")
Loading pt checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  5.51it/s]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  5.49it/s]

INFO 01-07 10:13:00 cpu_executor.py:186] # CPU blocks: 7281
INFO 01-07 10:13:00 llm_engine.py:431] init engine (profile, create kv cache, warmup model) took 0.08 seconds
INFO 01-07 10:13:02 api_server.py:640] Using supplied chat template:
INFO 01-07 10:13:02 api_server.py:640] None
INFO 01-07 10:13:02 launcher.py:19] Available routes are:
INFO 01-07 10:13:02 launcher.py:27] Route: /openapi.json, Methods: GET, HEAD
INFO 01-07 10:13:02 launcher.py:27] Route: /docs, Methods: GET, HEAD
INFO 01-07 10:13:02 launcher.py:27] Route: /docs/oauth2-redirect, Methods: GET, HEAD
INFO 01-07 10:13:02 launcher.py:27] Route: /redoc, Methods: GET, HEAD
INFO 01-07 10:13:02 launcher.py:27] Route: /health, Methods: GET
INFO 01-07 10:13:02 launcher.py:27] Route: /tokenize, Methods: POST
INFO 01-07 10:13:02 launcher.py:27] Route: /detokenize, Methods: POST
INFO 01-07 10:13:02 launcher.py:27] Route: /v1/models, Methods: GET
INFO 01-07 10:13:02 launcher.py:27] Route: /version, Methods: GET
INFO 01-07 10:13:02 launcher.py:27] Route: /v1/chat/completions, Methods: POST
INFO 01-07 10:13:02 launcher.py:27] Route: /v1/completions, Methods: POST
INFO 01-07 10:13:02 launcher.py:27] Route: /v1/embeddings, Methods: POST
INFO 01-07 10:13:02 launcher.py:27] Route: /pooling, Methods: POST
INFO 01-07 10:13:02 launcher.py:27] Route: /score, Methods: POST
INFO 01-07 10:13:02 launcher.py:27] Route: /v1/score, Methods: POST
INFO:     Started server process [1552]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8005 (Press CTRL+C to quit)

[Hoipang]

What image are you using ? Can you try my image hoipangg/vllm-cpu:v0.6.6.post2

Just tried by installing GDB too. It still worked in a standalone docker container.

root@1adb4ffd1855:/workspace# apt-get install gdb
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libbabeltrace1 libboost-regex1.74.0 libc6-dbg libdebuginfod-common libdebuginfod1 libdw1 libpython3.11 libsource-highlight-common libsource-highlight4v5
  sensible-utils ucf
Suggested packages:
  gdb-doc gdbserver
The following NEW packages will be installed:
  gdb libbabeltrace1 libboost-regex1.74.0 libc6-dbg libdebuginfod-common libdebuginfod1 libdw1 libpython3.11 libsource-highlight-common libsource-highlight4v5
  sensible-utils ucf
0 upgraded, 12 newly installed, 0 to remove and 0 not upgraded.
Need to get 15.5 MB of archives.
After this operation, 42.3 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://deb.debian.org/debian bookworm/main ppc64el sensible-utils all 0.0.17+nmu1 [19.0 kB]
Get:2 http://deb.debian.org/debian bookworm/main ppc64el ucf all 3.0043+nmu1 [55.2 kB]
Get:3 http://deb.debian.org/debian bookworm/main ppc64el libdebuginfod-common all 0.188-2.1 [21.5 kB]
Get:4 http://deb.debian.org/debian bookworm/main ppc64el libdw1 ppc64el 0.188-2.1 [253 kB]
Get:5 http://deb.debian.org/debian bookworm/main ppc64el libbabeltrace1 ppc64el 1.5.11-1+b2 [189 kB]
Get:6 http://deb.debian.org/debian bookworm/main ppc64el libdebuginfod1 ppc64el 0.188-2.1 [28.7 kB]
Get:7 http://deb.debian.org/debian bookworm/main ppc64el libpython3.11 ppc64el 3.11.2-6+deb12u4 [2085 kB]
Get:8 http://deb.debian.org/debian bookworm/main ppc64el libsource-highlight-common all 3.1.9-4.2 [77.4 kB]
Get:9 http://deb.debian.org/debian bookworm/main ppc64el libboost-regex1.74.0 ppc64el 1.74.0+ds1-21 [487 kB]
Get:10 http://deb.debian.org/debian bookworm/main ppc64el libsource-highlight4v5 ppc64el 3.1.9-4.2+b3 [262 kB]
Get:11 http://deb.debian.org/debian bookworm/main ppc64el gdb ppc64el 13.1-3 [3926 kB]
Get:12 http://deb.debian.org/debian bookworm/main ppc64el libc6-dbg ppc64el 2.36-9+deb12u9 [8133 kB]
Fetched 15.5 MB in 1s (13.9 MB/s)    
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package sensible-utils.
(Reading database ... 27315 files and directories currently installed.)
Preparing to unpack .../00-sensible-utils_0.0.17+nmu1_all.deb ...
Unpacking sensible-utils (0.0.17+nmu1) ...
Selecting previously unselected package ucf.
Preparing to unpack .../01-ucf_3.0043+nmu1_all.deb ...
Moving old data out of the way
Unpacking ucf (3.0043+nmu1) ...
Selecting previously unselected package libdebuginfod-common.
Preparing to unpack .../02-libdebuginfod-common_0.188-2.1_all.deb ...
Unpacking libdebuginfod-common (0.188-2.1) ...
Selecting previously unselected package libdw1:ppc64el.
Preparing to unpack .../03-libdw1_0.188-2.1_ppc64el.deb ...
Unpacking libdw1:ppc64el (0.188-2.1) ...
Selecting previously unselected package libbabeltrace1:ppc64el.
Preparing to unpack .../04-libbabeltrace1_1.5.11-1+b2_ppc64el.deb ...
Unpacking libbabeltrace1:ppc64el (1.5.11-1+b2) ...
Selecting previously unselected package libdebuginfod1:ppc64el.
Preparing to unpack .../05-libdebuginfod1_0.188-2.1_ppc64el.deb ...
Unpacking libdebuginfod1:ppc64el (0.188-2.1) ...
Selecting previously unselected package libpython3.11:ppc64el.
Preparing to unpack .../06-libpython3.11_3.11.2-6+deb12u4_ppc64el.deb ...
Unpacking libpython3.11:ppc64el (3.11.2-6+deb12u4) ...
Selecting previously unselected package libsource-highlight-common.
Preparing to unpack .../07-libsource-highlight-common_3.1.9-4.2_all.deb ...
Unpacking libsource-highlight-common (3.1.9-4.2) ...
Selecting previously unselected package libboost-regex1.74.0:ppc64el.
Preparing to unpack .../08-libboost-regex1.74.0_1.74.0+ds1-21_ppc64el.deb ...
Unpacking libboost-regex1.74.0:ppc64el (1.74.0+ds1-21) ...
Selecting previously unselected package libsource-highlight4v5:ppc64el.
Preparing to unpack .../09-libsource-highlight4v5_3.1.9-4.2+b3_ppc64el.deb ...
Unpacking libsource-highlight4v5:ppc64el (3.1.9-4.2+b3) ...
Selecting previously unselected package gdb.
Preparing to unpack .../10-gdb_13.1-3_ppc64el.deb ...
Unpacking gdb (13.1-3) ...
Selecting previously unselected package libc6-dbg:ppc64el.
Preparing to unpack .../11-libc6-dbg_2.36-9+deb12u9_ppc64el.deb ...
Unpacking libc6-dbg:ppc64el (2.36-9+deb12u9) ...
Setting up libdw1:ppc64el (0.188-2.1) ...
Setting up libpython3.11:ppc64el (3.11.2-6+deb12u4) ...
Setting up libsource-highlight-common (3.1.9-4.2) ...
Setting up libc6-dbg:ppc64el (2.36-9+deb12u9) ...
Setting up libboost-regex1.74.0:ppc64el (1.74.0+ds1-21) ...
Setting up libbabeltrace1:ppc64el (1.5.11-1+b2) ...
Setting up sensible-utils (0.0.17+nmu1) ...
Setting up libsource-highlight4v5:ppc64el (3.1.9-4.2+b3) ...
Setting up ucf (3.0043+nmu1) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 78.)
debconf: falling back to frontend: Readline
Setting up libdebuginfod-common (0.188-2.1) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 78.)
debconf: falling back to frontend: Readline
Setting up libdebuginfod1:ppc64el (0.188-2.1) ...
Setting up gdb (13.1-3) ...
Processing triggers for libc-bin (2.36-9+deb12u9) ...
root@1adb4ffd1855:/workspace# 
root@1adb4ffd1855:/workspace# 
root@1adb4ffd1855:/workspace# 
root@1adb4ffd1855:/workspace# 
root@1adb4ffd1855:/workspace# 
root@1adb4ffd1855:/workspace# /opt/conda/bin/python3 -m vllm.entrypoints.openai.api_server
INFO 01-07 10:12:37 api_server.py:712] vLLM API server version 0.6.6.post2.dev29+g6036d4db.d20250103
INFO 01-07 10:12:37 api_server.py:714] args: Namespace(host=None, port=8005, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='facebook/opt-125m', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='xgrammar', logits_processor_pattern=None, distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False)
INFO 01-07 10:12:37 api_server.py:199] Started engine process with PID 1587
INFO 01-07 10:12:47 config.py:2249] For POWERPC, we cast models to bfloat16 instead of using float16 by default. Float16 is not currently supported for POWERPC.
WARNING 01-07 10:12:47 config.py:2282] Casting torch.float16 to torch.bfloat16.
INFO 01-07 10:12:52 config.py:2249] For POWERPC, we cast models to bfloat16 instead of using float16 by default. Float16 is not currently supported for POWERPC.
WARNING 01-07 10:12:52 config.py:2282] Casting torch.float16 to torch.bfloat16.
INFO 01-07 10:12:54 config.py:516] This model supports multiple tasks: {'score', 'generate', 'classify', 'embed', 'reward'}. Defaulting to 'generate'.
WARNING 01-07 10:12:54 config.py:648] Async output processing is not supported on the current platform type cpu.
WARNING 01-07 10:12:54 cpu.py:56] CUDA graph is not supported on CPU, fallback to the eager mode.
WARNING 01-07 10:12:54 cpu.py:71] Environment variable VLLM_CPU_KVCACHE_SPACE (GB) for CPU backend is not set, using 4 by default.
INFO 01-07 10:12:58 config.py:516] This model supports multiple tasks: {'score', 'embed', 'classify', 'reward', 'generate'}. Defaulting to 'generate'.
WARNING 01-07 10:12:58 config.py:648] Async output processing is not supported on the current platform type cpu.
WARNING 01-07 10:12:58 cpu.py:56] CUDA graph is not supported on CPU, fallback to the eager mode.
WARNING 01-07 10:12:58 cpu.py:71] Environment variable VLLM_CPU_KVCACHE_SPACE (GB) for CPU backend is not set, using 4 by default.
INFO 01-07 10:12:58 importing.py:14] Triton not installed or not compatible; certain GPU-related functions will not be available.
INFO 01-07 10:12:58 llm_engine.py:234] Initializing an LLM engine (v0.6.6.post2.dev29+g6036d4db.d20250103) with config: model='facebook/opt-125m', speculative_config=None, tokenizer='facebook/opt-125m', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cpu, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=facebook/opt-125m, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"candidate_compile_sizes":[],"compile_sizes":[],"capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True, 
INFO 01-07 10:13:00 cpu.py:33] Cannot use _Backend.FLASH_ATTN backend on CPU.
INFO 01-07 10:13:00 selector.py:141] Using Torch SDPA backend.
INFO 01-07 10:13:00 weight_utils.py:251] Using model weights format ['*.bin']
Loading pt checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
/opt/conda/lib/python3.11/site-packages/vllm-0.6.6.post2.dev29+g6036d4db.d20250103.cpu-py3.11-linux-ppc64le.egg/vllm/model_executor/model_loader/weight_utils.py:450: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  state = torch.load(bin_file, map_location="cpu")
Loading pt checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  5.51it/s]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  5.49it/s]

INFO 01-07 10:13:00 cpu_executor.py:186] # CPU blocks: 7281
INFO 01-07 10:13:00 llm_engine.py:431] init engine (profile, create kv cache, warmup model) took 0.08 seconds
INFO 01-07 10:13:02 api_server.py:640] Using supplied chat template:
INFO 01-07 10:13:02 api_server.py:640] None
INFO 01-07 10:13:02 launcher.py:19] Available routes are:
INFO 01-07 10:13:02 launcher.py:27] Route: /openapi.json, Methods: GET, HEAD
INFO 01-07 10:13:02 launcher.py:27] Route: /docs, Methods: GET, HEAD
INFO 01-07 10:13:02 launcher.py:27] Route: /docs/oauth2-redirect, Methods: GET, HEAD
INFO 01-07 10:13:02 launcher.py:27] Route: /redoc, Methods: GET, HEAD
INFO 01-07 10:13:02 launcher.py:27] Route: /health, Methods: GET
INFO 01-07 10:13:02 launcher.py:27] Route: /tokenize, Methods: POST
INFO 01-07 10:13:02 launcher.py:27] Route: /detokenize, Methods: POST
INFO 01-07 10:13:02 launcher.py:27] Route: /v1/models, Methods: GET
INFO 01-07 10:13:02 launcher.py:27] Route: /version, Methods: GET
INFO 01-07 10:13:02 launcher.py:27] Route: /v1/chat/completions, Methods: POST
INFO 01-07 10:13:02 launcher.py:27] Route: /v1/completions, Methods: POST
INFO 01-07 10:13:02 launcher.py:27] Route: /v1/embeddings, Methods: POST
INFO 01-07 10:13:02 launcher.py:27] Route: /pooling, Methods: POST
INFO 01-07 10:13:02 launcher.py:27] Route: /score, Methods: POST
INFO 01-07 10:13:02 launcher.py:27] Route: /v1/score, Methods: POST
INFO:     Started server process [1552]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8005 (Press CTRL+C to quit)

[npanpaliya]

I've built an image using Dockerfile.ppc64le for ppc64le architecture.

[youkaichao]

Looks quite strange. I don't have any ideas right now.

[npanpaliya]

We found the root cause of this issue. The vllm image was built on Power10 and we ran it on Power9 which gave us illegal instruction error and the server couldn't start.

So, closing this issue.