[Feature] Implement InternVL to llama.cpp

[James4Ever0]

Motivation

Many llama.cpp users are requesting this so far. Ollama is one of the interfaces of llama.cpp and it is quite popular. Implementing it will significantly accelerate InterVL adoption and recognition.

Related resources

ggml-org/llama.cpp#6803

Additional context

InternVL is based on LLaMA architecture. Currently text-only InternLM models have been ported to Ollama but not for multimodal ones.

[G-z-w]

Thank you for your suggestions. We will gradually push support for various frameworks, and we also welcome contributions from the community.

[James4Ever0]

Hey @czczup would you please clarify the reason closing this issue?

[sammcj]

Any update on internVL support with llama.cpp?

[cloudyuyuyu]

Any update on internVL support with llama.cpp?

[G-z-w]

Thank you for your attention. We are actively progressing on this work, and we also welcome contributions from the community.

[rampageservices]

Just curious why this issue was closed if you are actively progressing on this work?

[rampageservices]

Thanks for reopening this issue.

…

On Mon, Sep 30, 2024 at 2:54 PM Zhe Chen ***@***.***> wrote: Reopened #522 <#522>. — Reply to this email directly, view it on GitHub <#522 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE32PCTXAB3YAMM2F3GTQLLZZDYT7AVCNFSM6AAAAABM2CFCKSVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJUGQ2TAMRXGIZDAMA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[cloudyuyuyu]

any update?

[Cartomex-MX]

November 12, any update? thank in advance

[BleedingDev]

Is there anything that could be helped with? :) InternVL2.5 is really important for the future. :)

[linxhome]

pay close attention to this, any update? pls

[James4Ever0]

Is it really that hard to do this? I suggest myself to implement it.

Current progress: ggml-org/llama.cpp#9403

Also ipex-llm has support for InternVL2.

[G-z-w]

Is it really that hard to do this? I suggest myself to implement it.

Thank you for your willingness to help! We greatly appreciate your initiative and would be glad to have your contributions. If you need us to provide any content or information, feel free to let us know.

[James4Ever0]

@G-z-w Is this model based on the LLaVA architecture? What are the differences, in input, output and internal parameters, and more?

[G-z-w]

@G-z-w Is this model based on the LLaVA architecture? What are the differences, in input, output and internal parameters, and more?

This model, along with other InternVL chat models, is similar to the LLaVA framework, with the specific structure shown in the link. The difference lies in dynamic resolution and pixel shuffle.

If convenient, we recommend prioritizing the deployment of the InternVL 2.5 series. The details of parameters are in model card of blog.

[James4Ever0]

@G-z-w Is this model based on the LLaVA architecture? What are the differences, in input, output and internal parameters, and more?

This model, along with other InternVL chat models, is similar to the LLaVA framework, with the specific structure shown in the link. The difference lies in dynamic resolution and pixel shuffle.

If convenient, we recommend prioritizing the deployment of the InternVL 2.5 series. The details of parameters are in model card of blog.

Is the model structure of v2.5 identical to v1.5 series? Now I can run v1.5 on llama.cpp. qlylangyu/llama.cpp#1

[G-z-w]

@G-z-w Is this model based on the LLaVA architecture? What are the differences, in input, output and internal parameters, and more?

This model, along with other InternVL chat models, is similar to the LLaVA framework, with the specific structure shown in the link. The difference lies in dynamic resolution and pixel shuffle.
If convenient, we recommend prioritizing the deployment of the InternVL 2.5 series. The details of parameters are in model card of blog.

Is the model structure of v2.5 identical to v1.5 series? Now I can run v1.5 on llama.cpp.

Yes, the structure of v2.5 is identical to that of the v1.5 series, except that the v2.5 series use different language models.

[BakingBrains]

@James4Ever0 any luck in running v2.5 model?

Thank you

[gryffindor-rr]

I tried in llama.cpp today, still not supported yet:
python3 convert_hf_to_gguf.py model_path
ERROR:hf-to-gguf:Model InternVLChatModel is not supported

Any update for luck?

[MrZeros]

any update?

[James4Ever0]

For anyone who is about to work with the current code, you could check my latest release here.

The 18 KB archive contains function-level diffs using universal ctags and some Python magic so anyone with entry-level C++ knowledge should be able to merge the changes easily.

[BrandonJull]

+1