[Installation] pip install vllm (0.6.3) will force a reinstallation of the CPU version torch and replace cuda torch on windows

[xiezhipeng-git]

pip install vllm (0.6.3) will force a reinstallation of the CPU version torch and replace cuda torch on windows. pip install vllm（0.6.3）将强制重新安装CPU版本的torch并在Windows上替换cuda torch。

I don't quite get what you mean, how can you have different versions of torch for CPU and GPU at the same time?我不太明白你的意思，你怎么能有不同版本的火炬CPU和GPU在同一时间？

only cuda torch

 pip install vllm --no-deps
Collecting vllm
  Using cached vllm-0.6.3.post1.tar.gz (2.7 MB)
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 2
  ╰─> [86 lines of output]
      Collecting cmake>=3.26
        Using cached cmake-3.30.5-py3-none-win_amd64.whl.metadata (6.4 kB)
      Collecting ninja
        Using cached ninja-1.11.1.1-py2.py3-none-win_amd64.whl.metadata (5.4 kB)

      Collecting packaging
        Using cached packaging-24.1-py3-none-any.whl.metadata (3.2 kB)
      Collecting setuptools>=61
        Using cached setuptools-75.2.0-py3-none-any.whl.metadata (6.9 kB)
      Collecting setuptools-scm>=8.0
        Using cached setuptools_scm-8.1.0-py3-none-any.whl.metadata (6.6 kB)
      Collecting torch==2.4.0
        Using cached torch-2.4.0-cp310-cp310-win_amd64.whl.metadata (27 kB)
      Collecting wheel
        Using cached wheel-0.44.0-py3-none-any.whl.metadata (2.3 kB)
      Collecting jinja2
        Using cached jinja2-3.1.4-py3-none-any.whl.metadata (2.6 kB)
      Collecting filelock (from torch==2.4.0)
        Using cached filelock-3.16.1-py3-none-any.whl.metadata (2.9 kB)
      Collecting typing-extensions>=4.8.0 (from torch==2.4.0)
        Using cached typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)

      Collecting sympy (from torch==2.4.0)
        Using cached sympy-1.13.3-py3-none-any.whl.metadata (12 kB)
      Collecting networkx (from torch==2.4.0)
        Using cached networkx-3.4.2-py3-none-any.whl.metadata (6.3 kB)
      Collecting fsspec (from torch==2.4.0)
        Using cached fsspec-2024.10.0-py3-none-any.whl.metadata (11 kB)
      Collecting tomli>=1 (from setuptools-scm>=8.0)
        Using cached tomli-2.0.2-py3-none-any.whl.metadata (10.0 kB)
      Collecting MarkupSafe>=2.0 (from jinja2)
        Using cached MarkupSafe-3.0.2-cp310-cp310-win_amd64.whl.metadata (4.1 kB
)
      Collecting mpmath<1.4,>=1.1.0 (from sympy->torch==2.4.0)
        Using cached mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB)
      Downloading torch-2.4.0-cp310-cp310-win_amd64.whl (197.9 MB)
                                                  3.9/197.9 MB 21.3 kB/s eta 2:3
1:31
      ERROR: Exception:
      Traceback (most recent call last):
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_vendor\urllib3\resp
onse.py", line 438, in _error_catcher
          yield
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_vendor\urllib3\resp
onse.py", line 561, in read
          data = self._fp_read(amt) if not fp_closed else b""
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_vendor\urllib3\resp
onse.py", line 527, in _fp_read
          return self._fp.read(amt) if amt is not None else self._fp.read()
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_vendor\cachecontrol
\filewrapper.py", line 98, in read
          data: bytes = self.__fp.read(amt)
        File "D:\my\env\python3.10.10\lib\http\client.py", line 465, in read
          s = self.fp.read(amt)
        File "D:\my\env\python3.10.10\lib\socket.py", line 705, in readinto
          return self._sock.recv_into(b)
        File "D:\my\env\python3.10.10\lib\ssl.py", line 1274, in recv_into
          return self.read(nbytes, buffer)
        File "D:\my\env\python3.10.10\lib\ssl.py", line 1130, in read
          return self._sslobj.read(len, buffer)
      TimeoutError: The read operation timed out
     
      During handling of the above exception, another exception occurred:
     
      Traceback (most recent call last):
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\cli\base_c
ommand.py", line 105, in _run_wrapper
          status = _inner_run()
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\cli\base_c
ommand.py", line 96, in _inner_run
          return self.run(options, args)
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\cli\req_co
mmand.py", line 67, in wrapper
          return func(self, options, args)
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\commands\i
nstall.py", line 379, in run
          requirement_set = resolver.resolve(
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\resolution
\resolvelib\resolver.py", line 179, in resolve
          self.factory.preparer.prepare_linked_requirements_more(reqs)
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\operations
\prepare.py", line 554, in prepare_linked_requirements_more
          self._complete_partial_requirements(
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\operations
\prepare.py", line 469, in _complete_partial_requirements
          for link, (filepath, _) in batch_download:
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\network\do
wnload.py", line 184, in __call__
          for chunk in chunks:
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\cli\progre
ss_bars.py", line 55, in _rich_progress_bar
          for chunk in iterable:
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\network\ut
ils.py", line 65, in response_chunks
          for chunk in response.raw.stream(
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_vendor\urllib3\resp
onse.py", line 622, in stream
          data = self.read(amt=amt, decode_content=decode_content)
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_vendor\urllib3\resp
onse.py", line 560, in read
          with self._error_catcher():
        File "D:\my\env\python3.10.10\lib\contextlib.py", line 153, in __exit__
          self.gen.throw(typ, value, traceback)
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_vendor\urllib3\resp
onse.py", line 443, in _error_catcher
          raise ReadTimeoutError(self._pool, None, "Read timed out.")
      pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host=
'files.pythonhosted.org', port=443): Read timed out.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem wit
h pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 2
╰─> See above for output.

If you internet is not good. You are so lucky. Because it will fail during the process of forcibly replacing CUDA torch with CPU. If you have a good internet connection. So things will become very bad. Your torch will transition from CUDA to a lower version CPU. And pip install vllm --no-deps or pip install vllm has same issue

What is your original version of pytorch?

Originally posted by @DarkLight1337 in #4194 (comment)

pip show torch
Name: torch
Version: 2.5.0+cu124
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
Home-page: https://pytorch.org/
Author: PyTorch Team
Author-email: packages@pytorch.org
License: BSD-3-Clause
Location: d:\my\env\python3.10.10\lib\site-packages
Requires: filelock, fsspec, jinja2, networkx, sympy, typing-extensions
Required-by: accelerate, auto_gptq, bitsandbytes, compressed-tensors, encodec, flash_attn, optimum, peft, stable-baselines3, timm, t
orchaudio, torchvision, trl, vector-quantize-pytorch, vocos

[xiezhipeng-git]

@DarkLight1337

[DarkLight1337]

Please follow these instructions on how to install for CPU.

[xiezhipeng-git]

Please follow these instructions on how to install for CPU.

why install cpu？and you say cpu is vllm？this function can use in windows?
and this  pip install -v -r requirements-cpu.txt --extra-index-url https://download.pytorch.org/whl/cpu
will replace torch? It is error.

[DarkLight1337]

Please follow these instructions on how to install for CPU.

why install cpu？and you say cpu is vllm？this function can use in windows?
and this  pip install -v -r requirements-cpu.txt --extra-index-url https://download.pytorch.org/whl/cpu
will replace torch? It is error.

Based on your title, you originally have PyTorch for CPU installed and do not want the CUDA version to be installed, so I guess you want the CPU version of vLLM as well. Correct me if I'm wrong.

[DarkLight1337]

On Windows, you should be able to use vLLM via WSL if I recall correctly.

[xiezhipeng-git]

Please follow these instructions on how to install for CPU.请按照以下说明安装CPU。

why install cpu？and you say cpu is vllm？this function can use in windows?
and this  pip install -v -r requirements-cpu.txt --extra-index-url https://download.pytorch.org/whl/cpu
will replace torch? It is error.

Based on your title, you originally have PyTorch for CPU installed and do not want the CUDA version to be installed, so I guess you want the CPU version of vLLM as well. Correct me if I'm wrong.根据你的标题，你最初安装了PyTorch for CPU，不想安装CUDA版本，所以我猜你也想要vLLM的CPU版本。如果我说错了请纠正我。

Of course not. It is said from beginning to end that the CUDA version has been replaced with the CPU version. Of course, all I want is the CUDA version。
如果你看得懂中文的话。还是用 中文交流吧。省的再有误解。从头到尾都说的是cuda版本被cpu替换了。并且我还说变成cpu版本这非常糟糕。只能重装cuda版本。中国的网络环境，这非常浪费时间。只要cuda版本
从来没有一次装过cpu版本的vllm .都是vllm 出错导致cuda版本被替换成了cpu。标题也是一个意思。
如果可能我不希望在使用wsl了。因为wsl有很大的概率导致会有未知情况下虚拟机崩溃后无法再次启动。然后整个虚拟机就没法用了。找wsl开发团队也没办法解决。100多个G的内容只能删掉了。完全浪费青春

[DarkLight1337]

Please follow these instructions on how to install for CPU.请按照以下说明安装CPU。

why install cpu？and you say cpu is vllm？this function can use in windows?
and this  pip install -v -r requirements-cpu.txt --extra-index-url https://download.pytorch.org/whl/cpu
will replace torch? It is error.

Based on your title, you originally have PyTorch for CPU installed and do not want the CUDA version to be installed, so I guess you want the CPU version of vLLM as well. Correct me if I'm wrong.根据你的标题，你最初安装了PyTorch for CPU，不想安装CUDA版本，所以我猜你也想要vLLM的CPU版本。如果我说错了请纠正我。

Of course not. It is said from beginning to end that the CUDA version has been replaced with the CPU version. Of course, all I want is the CUDA version。 如果你看得懂中文的话。还是用 中文交流吧。省的再有误解。从头到尾都说的是cuda版本被cpu替换了。并且我还说变成cpu版本这非常糟糕。只能重装cuda版本。中国的网络环境，这非常浪费时间。只要cuda版本

Oh sorry, I somehow read it the other way round. vLLM only officially supports Linux OS so it might not be able to detect your CUDA from native Windows. I suggest using vLLM through WSL.

[xiezhipeng-git]

Please follow these instructions on how to install for CPU.请按照以下说明安装CPU。

why install cpu？and you say cpu is vllm？this function can use in windows?
and this  pip install -v -r requirements-cpu.txt --extra-index-url https://download.pytorch.org/whl/cpu
will replace torch? It is error.

Based on your title, you originally have PyTorch for CPU installed and do not want the CUDA version to be installed, so I guess you want the CPU version of vLLM as well. Correct me if I'm wrong.根据你的标题，你最初安装了PyTorch for CPU，不想安装CUDA版本，所以我猜你也想要vLLM的CPU版本。如果我说错了请纠正我。

Of course not. It is said from beginning to end that the CUDA version has been replaced with the CPU version. Of course, all I want is the CUDA version。 如果你看得懂中文的话。还是用 中文交流吧。省的再有误解。从头到尾都说的是cuda版本被cpu替换了。并且我还说变成cpu版本这非常糟糕。只能重装cuda版本。中国的网络环境，这非常浪费时间。只要cuda版本

Oh sorry, I somehow read it the other way round. vLLM only officially supports Linux OS so it might not be able to detect your CUDA from native Windows. I suggest using vLLM through WSL.

wsl 有致命缺陷。如果可以我都不在愿意使用wsl了。你的问题检测不到torch cuda。这听起来像是pip 和cmake命令没有处理好。可以学习一下flashattntion .flashattntion 可以在windows上安装了。通过 不隔离环境 就可以找到torch 并编译安装。如果问题是这样的话，应该是不难解决的。
WSL has a fatal flaw. If possible, I am no longer willing to use WSL. Your issue cannot detect torch CUDA. This sounds like the pip and cmake commands were not handled properly. You can learn Flashatttion. Flashhattntion can be installed on Windows now. By not isolating the environment, torch can be found and compiled for installation. If that's the case, it shouldn't be difficult to solve.

[DarkLight1337]

@dtrifiro what's your opinion on supporting Windows? Is it feasible at this stage?

[xiezhipeng-git]

@DarkLight1337 @dtrifiro
而且之所以torch版本被替换成cpu .估计是因为你们对torch的版本需求写的是==2.4.0 。改成>=2.4.0.应该就不会有被自动重装的问题了。然后在考虑看看能否正式支持windows
And the reason why the torch version was replaced with cpu .Probably because you guys wrote ==2.4.0 for the torch version requirement .Change it to >=2.4.0. and you shouldn't have the problem of being reinstalled automatically.Then we'll see if we can officially support windows.

[DarkLight1337]

@DarkLight1337 @dtrifiro 而且之所以torch版本被替换成cpu .估计是因为你们对torch的版本需求写的是==2.4.0 。改成>=2.4.0.应该就不会有被自动重装的问题了。然后在考虑看看能否正式支持windows And the reason why the torch version was replaced with cpu .Probably because you guys wrote ==2.4.0 for the torch version requirement .Change it to >=2.4.0. and you shouldn't have the problem of being reinstalled automatically.Then we'll see if we can officially support windows.

From my understanding, PyTorch installation should be able to automatically choose CPU/CUDA based on your machine. What happens if you just install torch==2.4.0 directly?

[xiezhipeng-git]

中国区网络环境不好。不想在折腾torch的版本。之前使用pip install vllm 安装成功以后。发现了torch 被替换成了cpu的这个问题。不过在装完cuda的torch以后我运行过一次vllm。然后报错了， 错误如下（不是很确定后来这个错误有没有被覆盖）
WARNING 10-24 21:42:41 _custom_ops.py:19] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
ERROR 10-24 21:42:49 registry.py:267] Error in inspecting model architecture 'Qwen2ForCausalLM'
ERROR 10-24 21:42:49 registry.py:267] Traceback (most recent call last):
ERROR 10-24 21:42:49 registry.py:267] File "d:\my\env\python3.10.10\lib\site-packages\vllm\model_executor\models\registry.py", line 429, in _run_in_subprocess
ERROR 10-24 21:42:49 registry.py:267] returned.check_returncode()
ERROR 10-24 21:42:49 registry.py:267] File "d:\my\env\python3.10.10\lib\subprocess.py", line 457, in check_returncode
ERROR 10-24 21:42:49 registry.py:267] raise CalledProcessError(self.returncode, self.args, self.stdout,
ERROR 10-24 21:42:49 registry.py:267] subprocess.CalledProcessError: Command '['d:\my\env\python3.10.10\python.exe', '-m', 'vllm.model_executor.models.registry']' returned non-zero exit status 1.
ERROR 10-24 21:42:49 registry.py:267]
ERROR 10-24 21:42:49 registry.py:267] The above exception was the direct cause of the following exception:
ERROR 10-24 21:42:49 registry.py:267]
ERROR 10-24 21:42:49 registry.py:267] Traceback (most recent call last):
ERROR 10-24 21:42:49 registry.py:267] File "d:\my\env\python3.10.10\lib\site-packages\vllm\model_executor\models\registry.py", line 265, in _try_inspect_model_cls
ERROR 10-24 21:42:49 registry.py:267] return model.inspect_model_cls()
ERROR 10-24 21:42:49 registry.py:267] File "d:\my\env\python3.10.10\lib\site-packages\vllm\model_executor\models\registry.py", line 227, in inspect_model_cls
ERROR 10-24 21:42:49 registry.py:267] return _run_in_subprocess(
ERROR 10-24 21:42:49 registry.py:267] File "d:\my\env\python3.10.10\lib\site-packages\vllm\model_executor\models\registry.py", line 432, in _run_in_subprocess
ERROR 10-24 21:42:49 registry.py:267] raise RuntimeError(f"Error raised in subprocess:\n"
ERROR 10-24 21:42:49 registry.py:267] RuntimeError: Error raised in subprocess:
ERROR 10-24 21:42:49 registry.py:267] d:\my\env\python3.10.10\lib\runpy.py:126: RuntimeWarning: 'vllm.model_executor.models.registry' found in sys.modules after import of package 'vllm.model_executor.models', but prior to execution of 'vllm.model_executor.models.registry'; this may result in unpredictable behaviour
ERROR 10-24 21:42:49 registry.py:267] warn(RuntimeWarning(msg))
ERROR 10-24 21:42:49 registry.py:267] Traceback (most recent call last):
ERROR 10-24 21:42:49 registry.py:267] File "d:\my\env\python3.10.10\lib\runpy.py", line 196, in _run_module_as_main
ERROR 10-24 21:42:49 registry.py:267] return _run_code(code, main_globals, None,
ERROR 10-24 21:42:49 registry.py:267] File "d:\my\env\python3.10.10\lib\runpy.py", line 86, in _run_code
ERROR 10-24 21:42:49 registry.py:267] exec(code, run_globals)
ERROR 10-24 21:42:49 registry.py:267] File "d:\my\env\python3.10.10\lib\site-packages\vllm\model_executor\models\registry.py", line 453, in
ERROR 10-24 21:42:49 registry.py:267] _run()
ERROR 10-24 21:42:49 registry.py:267] File "d:\my\env\python3.10.10\lib\site-packages\vllm\model_executor\models\registry.py", line 448, in _run
ERROR 10-24 21:42:49 registry.py:267] with open(output_file, "wb") as f:
ERROR 10-24 21:42:49 registry.py:267] PermissionError: [Errno 13] Permission denied: 'C:\Users\Admin\AppData\Local\Temp\tmp6cx7k05c'
ERROR 10-24 21:42:49 registry.py:267]
ValueError Traceback (most recent call last)
Cell In[2], line 5
1 from vllm import LLM, SamplingParams
3 # model_dir='Qwen2.5-14B-Instruct-GPTQ-Int4'
----> 5 llm = LLM(model=model_dir,enforce_eager=True)
6 sampling_params = SamplingParams( top_p=0.9, max_tokens=512,top_k=10)
8 prompt = "1+1等于几"

File d:\my\env\python3.10.10\lib\site-packages\vllm\entrypoints\llm.py:177, in LLM.init(self, model, tokenizer, tokenizer_mode, skip_tokenizer_init, trust_remote_code, tensor_parallel_size, dtype, quantization, revision, tokenizer_revision, seed, gpu_memory_utilization, swap_space, cpu_offload_gb, enforce_eager, max_context_len_to_capture, max_seq_len_to_capture, disable_custom_all_reduce, disable_async_output_proc, mm_processor_kwargs, **kwargs)
152 kwargs["disable_log_stats"] = True
154 engine_args = EngineArgs(
155 model=model,
156 tokenizer=tokenizer,
(...)
175 **kwargs,
176 )
--> 177 self.llm_engine = LLMEngine.from_engine_args(
178 engine_args, usage_context=UsageContext.LLM_CLASS)
179 self.request_counter = Counter()

File d:\my\env\python3.10.10\lib\site-packages\vllm\engine\llm_engine.py:570, in LLMEngine.from_engine_args(cls, engine_args, usage_context, stat_loggers)
568 """Creates an LLM engine from the engine arguments."""
569 # Create the engine configs.
--> 570 engine_config = engine_args.create_engine_config()
571 executor_class = cls._get_executor_cls(engine_config)
572 # Create the LLM engine.

File d:\my\env\python3.10.10\lib\site-packages\vllm\engine\arg_utils.py:903, in EngineArgs.create_engine_config(self)
898 assert self.cpu_offload_gb >= 0, (
899 "CPU offload space must be non-negative"
900 f", but got {self.cpu_offload_gb}")
902 device_config = DeviceConfig(device=self.device)
--> 903 model_config = self.create_model_config()
905 if model_config.is_multimodal_model:
906 if self.enable_prefix_caching:

File d:\my\env\python3.10.10\lib\site-packages\vllm\engine\arg_utils.py:839, in EngineArgs.create_model_config(self)
838 def create_model_config(self) -> ModelConfig:
--> 839 return ModelConfig(
840 model=self.model,
841 # We know this is not None because we set it in post_init
842 tokenizer=cast(str, self.tokenizer),
843 tokenizer_mode=self.tokenizer_mode,
844 trust_remote_code=self.trust_remote_code,
845 dtype=self.dtype,
846 seed=self.seed,
847 revision=self.revision,
848 code_revision=self.code_revision,
849 rope_scaling=self.rope_scaling,
850 rope_theta=self.rope_theta,
851 tokenizer_revision=self.tokenizer_revision,
852 max_model_len=self.max_model_len,
853 quantization=self.quantization,
854 quantization_param_path=self.quantization_param_path,
855 enforce_eager=self.enforce_eager,
856 max_context_len_to_capture=self.max_context_len_to_capture,
857 max_seq_len_to_capture=self.max_seq_len_to_capture,
858 max_logprobs=self.max_logprobs,
859 disable_sliding_window=self.disable_sliding_window,
860 skip_tokenizer_init=self.skip_tokenizer_init,
861 served_model_name=self.served_model_name,
862 limit_mm_per_prompt=self.limit_mm_per_prompt,
863 use_async_output_proc=not self.disable_async_output_proc,
864 override_neuron_config=self.override_neuron_config,
865 config_format=self.config_format,
866 mm_processor_kwargs=self.mm_processor_kwargs,
867 )

File d:\my\env\python3.10.10\lib\site-packages\vllm\config.py:200, in ModelConfig.init(self, model, tokenizer, tokenizer_mode, trust_remote_code, dtype, seed, revision, code_revision, rope_scaling, rope_theta, tokenizer_revision, max_model_len, spec_target_max_model_len, quantization, quantization_param_path, enforce_eager, max_context_len_to_capture, max_seq_len_to_capture, max_logprobs, disable_sliding_window, skip_tokenizer_init, served_model_name, limit_mm_per_prompt, use_async_output_proc, override_neuron_config, config_format, mm_processor_kwargs)
192 self.max_model_len = _get_and_verify_max_len(
193 hf_config=self.hf_text_config,
194 max_model_len=max_model_len,
195 disable_sliding_window=self.disable_sliding_window,
196 sliding_window_len=self.get_hf_config_sliding_window(),
197 spec_target_max_model_len=spec_target_max_model_len)
198 self.served_model_name = get_served_model_name(model,
199 served_model_name)
--> 200 self.multimodal_config = self._init_multimodal_config(
201 limit_mm_per_prompt)
202 if not self.skip_tokenizer_init:
203 self._verify_tokenizer_mode()

File d:\my\env\python3.10.10\lib\site-packages\vllm\config.py:219, in ModelConfig._init_multimodal_config(self, limit_mm_per_prompt)
215 def _init_multimodal_config(
216 self, limit_mm_per_prompt: Optional[Mapping[str, int]]
217 ) -> Optional["MultiModalConfig"]:
218 architectures = getattr(self.hf_config, "architectures", [])
--> 219 if ModelRegistry.is_multimodal_model(architectures):
220 return MultiModalConfig(limit_per_prompt=limit_mm_per_prompt or {})
222 if limit_mm_per_prompt:

File d:\my\env\python3.10.10\lib\site-packages\vllm\model_executor\models\registry.py:387, in _ModelRegistry.is_multimodal_model(self, architectures)
383 def is_multimodal_model(
384 self,
385 architectures: Union[str, List[str]],
386 ) -> bool:
--> 387 return self.inspect_model_cls(architectures).supports_multimodal

File d:\my\env\python3.10.10\lib\site-packages\vllm\model_executor\models\registry.py:356, in _ModelRegistry.inspect_model_cls(self, architectures)
353 if model_info is not None:
354 return model_info
--> 356 return self._raise_for_unsupported(architectures)

File d:\my\env\python3.10.10\lib\site-packages\vllm\model_executor\models\registry.py:317, in _ModelRegistry._raise_for_unsupported(self, architectures)
314 def _raise_for_unsupported(self, architectures: List[str]):
315 all_supported_archs = self.get_supported_archs()
--> 317 raise ValueError(
318 f"Model architectures {architectures} are not supported for now. "
319 f"Supported architectures: {all_supported_archs}")

ValueError: Model architectures ['Qwen2ForCausalLM'] are not supported for now. Supported architectures: ['AquilaModel', 'AquilaForCausalLM', 'ArcticForCausalLM', 'BaiChuanForCausalLM', 'BaichuanForCausalLM', 'BloomForCausalLM', 'CohereForCausalLM', 'DbrxForCausalLM', 'DeciLMForCausalLM', 'DeepseekForCausalLM', 'DeepseekV2ForCausalLM', 'ExaoneForCausalLM', 'FalconForCausalLM', 'GemmaForCausalLM', 'Gemma2ForCausalLM', 'GPT2LMHeadModel', 'GPTBigCodeForCausalLM', 'GPTJForCausalLM', 'GPTNeoXForCausalLM', 'GraniteForCausalLM', 'GraniteMoeForCausalLM', 'InternLMForCausalLM', 'InternLM2ForCausalLM', 'JAISLMHeadModel', 'JambaForCausalLM', 'LlamaForCausalLM', 'LLaMAForCausalLM', 'MambaForCausalLM', 'MistralForCausalLM', 'MixtralForCausalLM', 'QuantMixtralForCausalLM', 'MptForCausalLM', 'MPTForCausalLM', 'MiniCPMForCausalLM', 'MiniCPM3ForCausalLM', 'NemotronForCausalLM', 'OlmoForCausalLM', 'OlmoeForCausalLM', 'OPTForCausalLM', 'OrionForCausalLM', 'PersimmonForCausalLM', 'PhiForCausalLM', 'Phi3ForCausalLM', 'Phi3SmallForCausalLM', 'PhiMoEForCausalLM', 'Qwen2ForCausalLM', 'Qwen2MoeForCausalLM', 'RWForCausalLM', 'StableLMEpochForCausalLM', 'StableLmForCausalLM', 'Starcoder2ForCausalLM', 'SolarForCausalLM', 'XverseForCausalLM', 'BartModel', 'BartForConditionalGeneration', 'Gemma2Model', 'MistralModel', 'Qwen2ForRewardModel', 'Phi3VForCausalLM', 'Blip2ForConditionalGeneration', 'ChameleonForConditionalGeneration', 'ChatGLMModel', 'ChatGLMForConditionalGeneration', 'FuyuForCausalLM', 'InternVLChatModel', 'LlavaForConditionalGeneration', 'LlavaNextForConditionalGeneration', 'LlavaNextVideoForConditionalGeneration', 'LlavaOnevisionForConditionalGeneration', 'MiniCPMV', 'MolmoForCausalLM', 'NVLM_D', 'PaliGemmaForConditionalGeneration', 'PixtralForConditionalGeneration', 'QWenLMHeadModel', 'Qwen2VLForConditionalGeneration', 'UltravoxModel', 'MllamaForConditionalGeneration', 'EAGLEModel', 'MedusaModel', 'MLPSpeculatorPreTrainedModel']