Does it support multi-modal LLM like LLaVa?

[yizhengzhang1]

The given example shows how to use for LLM like llama.
But how can I use it to accelerate Visual Language Model like LLaVa?

[simon-mo]

LLaVa needs to be added with our lightweight model addition process. Contribution welcomed! https://docs.vllm.ai/en/latest/models/adding_model.html

[l4b4r4b4b4]

LLaVa needs to be added with our lightweight model addition process. Contribution welcomed! https://docs.vllm.ai/en/latest/models/adding_model.html

You dont generally see an issue in using the lighweight model adidtion for the composed llava model with CLIP, vision encoder and LLM?

[simon-mo]

It does turns out to be more complex. A working PR is here #3042

[ywang96]

Closing as #3042 adds support for LLaVA1.5

[iamsaurabhgupt]

is LLaVA-Next supported?
couldn't find a working example

[ywang96]

@iamsaurabhgupt Feel free to take a look at the example here! #4194 (comment). One limitation we have now is that we don't support dynamic shape of input image, so the results might be slightly different from huggingface implementation, but we're on track to support this very soon!