Add support for CogVLM model

[Tianyue-Zhao]

This addresses the requests for CogVLM in #4387 and #4350.
CogVLM is a pretty popular model that now adds in cleanly after the recent additions to libmtmd.
I've converted a GGUF here: Link to GGUF files

Sample command and output:

build/bin/llama-mtmd-cli -m ../cogvlm-chat-hf/cogvlm-13B-chat-v1.1-F16.gguf --mmproj ../cogvlm-chat-hf/mmproj-cogvlm-chat-hf --image ./community.png --chat-template vicuna -p "Describe the picture"

load_hparams: model size:         8448.53 MiB
load_hparams: metadata size:      0.36 MiB
alloc_compute_meta:        CPU compute buffer size =   142.02 MiB
main: loading model: ../cogvlm-chat-hf/cogvlm-13B-chat-v1.1-F16.gguf
encoding image slice...
image slice encoded in 16135 ms
decoding image batch 1/1, n_tokens_batch = 1227
image decoded (batch 1/1) in 54065 ms

1. The image showcases a futuristic urban landscape with a mix of architectural styles. The buildings are multi-storied and have a combination of traditional and modern elements. There's a prominent tree in the foreground, suggesting a blend of nature and urban development. The scene appears to be bustling with activity, with various signs and billboards, indicating commercial or residential zones.


llama_perf_context_print:        load time =  108969.65 ms
llama_perf_context_print: prompt eval time =   85229.27 ms /  1241 tokens (   68.68 ms per token,    14.56 tokens per second)
llama_perf_context_print:        eval time =   19843.15 ms /    83 runs   (  239.07 ms per token,     4.18 tokens per second)
llama_perf_context_print:       total time =  126951.23 ms /  1324 tokens
llama_perf_context_print:    graphs reused =          0

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

[Tianyue-Zhao]

I think I've fixed the typecheck and format check workflows that were failing before, can someone approve the workflows to run again?
Also, is there a way to run these Github workflows locally or without needing approval from a reviewer?
It would be good to run these CI/CD checks myself before posting the PR.

[CISC]

Also, is there a way to run these Github workflows locally or without needing approval from a reviewer? It would be good to run these CI/CD checks myself before posting the PR.

You can run flake8, pyright and editorconfig locally (or via IDE plugins), the build tests can be run manually with ctest.

[CISC]

This is not a complete review as I don't know enough about mtmd, just commenting...

[Tianyue-Zhao]

Also, is there a way to run these Github workflows locally or without needing approval from a reviewer? It would be good to run these CI/CD checks myself before posting the PR.

You can run flake8, pyright and editorconfig locally (or via IDE plugins), the build tests can be run manually with ctest.

Thanks for the info! That's something I've been wondering about for a while.

[CISC]

Further refinement (merge cont+reshape).

[CISC]

Further refinement (merge cont+reshape).

After #15662 we can avoid these altogether and just create 3D views.

[CISC]

Further refinement (merge cont+reshape).

After #15662 we can avoid these altogether and just create 3D views.

Merged, rebase and apply updated suggestions.

[Tianyue-Zhao]

Further refinement (merge cont+reshape).

After #15662 we can avoid these altogether and just create 3D views.

Merged, rebase and apply updated suggestions.

Thanks for the reminder, I've rebased it and removed the extra ggml_cont calls.

[ngxson]

sorry I missed the notification to review this. will have a look & push commits to resolve the conflicts

[ngxson]

I don't have enough VRAM to test the model right now, but I think the code should be good to merge (after CI passed)

Feel free to give it a try even after the PR is merged. In case there are bugs, we can make follow-up PRs to fix it.

[ngxson]

No idea why the ASAN test failed, probably just a random runtime issue. I'm re-running the CI

[CISC]

No idea why the ASAN test failed, probably just a random runtime issue. I'm re-running the CI

It's ccache getting poisoned somehow, I've yet to track down the reason, when it happens you have to find and delete all the caches on the branch + master (I've just deleted the master ones) and rerun.

[ngxson]

btw @Tianyue-Zhao , it seems like this implementation still use the legacy llava preprocessing and does not support dynamic resolution. is this expected?