[Feature] InternVL-Chat-V1.5 Support

[77h2l]

Motivation

does lmdeploy support the InternVL-Chat-V1.5 VL model now? Is it supoorted on the main branch?Thx

Related resources

No response

Additional context

No response

[lvhan028]

PR #1490 is supporting it

[77h2l]

thx for your reply, does this pr have been completely implemented yet, support quant now? since the size of the model is 25B, A10 gpu is not able to afford the memory cost.

[lvhan028]

quant is not supported now. We are planning to implement VLM quant in May.

[lvhan028]

One more thing, we focus on the LLM quantization, not the vision model
What's the size of the vision model?

[77h2l]

actually InternVL-Chat-V1.5 is a 25B model, it need about 46.5G mem to load the model as fp16 format, btw i thought InternVL-Chat-V1.5 and LMdeploy are both from shanghai ai lab job

[lvhan028]

architecture: InternViT-6B-448px-V1-5 + MLP + InternLM2-Chat-20B
I am afraid it cannot fit into A10 (24G) even though LLM weights are quantized into 4 bits.

[lvhan028]

Hi, @77h2l
This model is supported in the latest version v0.4.1

Regarding 4bit quantization and better TP, we are going to open another PRs to tackle them