Misc. bug: Obvious performance downgrade between Vulkan and CUDA backend.

[MaoJianwei]

Name and Version

PS F:\llm\llama-b6568-bin-win-vulkan-x64> .\llama-cli.exe --version
load_backend: loaded RPC backend from F:\llm\llama-b6568-bin-win-vulkan-x64\ggml-rpc.dll
[2025-09-25 00:14:31.678][info][16388] [huya-helper.cpp:378#init_log] graphic-hook 64bit log init suceed.
exe:llama-cli.exe, pid:5032
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = Quadro P620 (NVIDIA) | uma: 0 | fp16: 0 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none
ggml_vulkan: 1 = Intel(R) UHD Graphics 630 (Intel Corporation) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 32768 | int dot: 0 | matrix cores: none
load_backend: loaded Vulkan backend from F:\llm\llama-b6568-bin-win-vulkan-x64\ggml-vulkan.dll
load_backend: loaded CPU backend from F:\llm\llama-b6568-bin-win-vulkan-x64\ggml-cpu-haswell.dll
version: 6568 (f2a789e3)
built with clang version 19.1.5 for x86_64-pc-windows-msvc

PS F:\llm\llama-b6568-bin-win-cuda-12.4-x64> .\llama-cli.exe --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: Quadro P620, compute capability 6.1, VMM: yes
load_backend: loaded CUDA backend from F:\llm\llama-b6568-bin-win-cuda-12.4-x64\ggml-cuda.dll
load_backend: loaded RPC backend from F:\llm\llama-b6568-bin-win-cuda-12.4-x64\ggml-rpc.dll
load_backend: loaded CPU backend from F:\llm\llama-b6568-bin-win-cuda-12.4-x64\ggml-cpu-haswell.dll
version: 6568 (f2a789e3)
built with clang version 19.1.5 for x86_64-pc-windows-msvc

Operating systems

Windows

Which llama.cpp modules do you know to be affected?

llama-server

Problem description & steps to reproduce

Run and compare the prompt eval and eval speed between vulkan and cuda backends.

Vulkan:
84.29 tokens per second for prefill phase
34.19 tokens per second for decode phase

CUDA:
524.58 tokens per second for prefill phase
17.96 tokens per second for decode phase

Why dose the performance difference exist?
Could you please improve Vulkan's prefill speed and CUDA's decode speed?

@JohannesGaessler @slaren @0cc4m

PS F:\llm\llama-b6568-bin-win-vulkan-x64> .\llama-server.exe --model ..\Qwen2.5-1.5B-Instruct-Q4_K_M.gguf  --no-mmap --jinja --verbose-prompt    --host 0.0.0.0  --flash-attn on --cache-type-k f32 --cache-type-v f32 -ngl 100 --metrics

PS F:\llm\llama-b6568-bin-win-cuda-12.4-x64> .\llama-server.exe --model ..\Qwen2.5-1.5B-Instruct-Q4_K_M.gguf  --no-mmap --jinja --verbose-prompt    --host 0.0.0.0  --flash-attn on --cache-type-k f32 --cache-type-v f32 -ngl 100 --metrics

First Bad Commit

No response

Relevant log output

PS F:\llm\llama-b6568-bin-win-vulkan-x64> .\llama-server.exe --model ..\Qwen2.5-1.5B-Instruct-Q4_K_M.gguf  --no-mmap --jinja --verbose-prompt    --host 0.0.0.0  --flash-attn on --cache-type-k f32 --cache-type-v f32 -ngl 100 --metrics


main: server is listening on http://0.0.0.0:8080 - starting the main loop
srv  update_slots: all slots are idle
srv  params_from_: Chat format: Hermes 2 Pro
slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 766
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 766, n_tokens = 766, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_past = 766, n_tokens = 766
slot      release: id  0 | task 0 | stop processing: n_past = 1459, truncated = 0
slot print_timing: id  0 | task 0 |
prompt eval time =    9087.16 ms /   766 tokens (   11.86 ms per token,    84.29 tokens per second)
       eval time =   20297.20 ms /   694 tokens (   29.25 ms per token,    34.19 tokens per second)
      total time =   29384.35 ms /  1460 tokens
srv  update_slots: all slots are idle

PS F:\llm\llama-b6568-bin-win-cuda-12.4-x64> .\llama-server.exe --model ..\Qwen2.5-1.5B-Instruct-Q4_K_M.gguf  --no-mmap --jinja --verbose-prompt    --host 0.0.0.0  --flash-attn on --cache-type-k f32 --cache-type-v f32 -ngl 100 --metrics


main: server is listening on http://0.0.0.0:8080 - starting the main loop
srv  update_slots: all slots are idle
srv  params_from_: Chat format: Hermes 2 Pro
slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 766
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 766, n_tokens = 766, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_past = 766, n_tokens = 766
slot      release: id  0 | task 0 | stop processing: n_past = 1352, truncated = 0
slot print_timing: id  0 | task 0 |
prompt eval time =    1460.21 ms /   766 tokens (    1.91 ms per token,   524.58 tokens per second)
       eval time =   32675.95 ms /   587 tokens (   55.67 ms per token,    17.96 tokens per second)
      total time =   34136.16 ms /  1353 tokens
srv  update_slots: all slots are idle

[taronaeo]

Is this a duplicate of #15955?

[JohannesGaessler]

IIRC I get the same token generation speed with my P40 on Linux regardless of the backend, could be a Windows issue.

[MaoJianwei]

Is this a duplicate of #15955?

yes, I didn't find the previous issue link just now :)

[jeffbolznv]

Please try setting the env var GGML_VK_VISIBLE_DEVICES=0 for Vulkan. I think your model will be split across the two GPUs by default.

[MaoJianwei]

Please try setting the env var GGML_VK_VISIBLE_DEVICES=0 for Vulkan. I think your model will be split across the two GPUs by default.

I will have a try. But I did't see the utilization ratio of the Device 1 (Intel integrated GPU) raised, it is about 0%.

[taronaeo]

I've ran a benchmark comparison between Vulkan and CUDA here:

llama-bench

Vulkan

$ build/bin/llama-bench -m /opt/hf_models/qwen2.5-1.5b-instruct-q4_k_m.gguf -t 8 -fa 1 -ngl 99

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 2060 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | --------------: | -------------------: |
| qwen2 1.5B Q4_K - Medium       |   1.04 GiB |     1.78 B | Vulkan     |  99 |  1 |           pp512 |     4017.99 ± 199.97 |
| qwen2 1.5B Q4_K - Medium       |   1.04 GiB |     1.78 B | Vulkan     |  99 |  1 |           tg128 |        163.72 ± 0.32 |

build: e7a5130a2 (6573)
Segmentation fault (core dumped)

CUDA

$ build/bin/llama-bench -m /opt/hf_models/qwen2.5-1.5b-instruct-q4_k_m.gguf -t 8 -fa 1 -ngl 99
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2060, compute capability 7.5, VMM: yes
| model                          |       size |     params | backend    | ngl | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | --------------: | -------------------: |
| qwen2 1.5B Q4_K - Medium       |   1.04 GiB |     1.78 B | CUDA       |  99 |  1 |           pp512 |     5391.85 ± 364.88 |
| qwen2 1.5B Q4_K - Medium       |   1.04 GiB |     1.78 B | CUDA       |  99 |  1 |           tg128 |        162.81 ± 0.37 |

build: e7a5130a2 (6573)

llama-cli

Vulkan

$ build/bin/llama-cli -m /opt/hf_models/qwen2.5-1.5b-instruct-q4_k_m.gguf -t 8 --no-mmap -fa on --cache-type-k f32 --cache-type-v f32 -ngl 99 -no-cnv -p "Write me a dog walking business idea 1. " -n 50 --seed 2362421855

Write me a dog walking business idea 1. 2. 3.

Sure, here are three dog walking business ideas for you:

1. **Doggy Digs**: This business focuses on creating customized dog digging experiences. You could create themed dog parks or play areas with specific digging areas, where

llama_perf_sampler_print:    sampling time =       4.84 ms /    61 runs   (    0.08 ms per token, 12590.30 tokens per second)
llama_perf_context_print:        load time =     931.24 ms
llama_perf_context_print: prompt eval time =      53.26 ms /    11 tokens (    4.84 ms per token,   206.55 tokens per second)
llama_perf_context_print:        eval time =     335.31 ms /    49 runs   (    6.84 ms per token,   146.13 tokens per second)
llama_perf_context_print:       total time =     417.37 ms /    60 tokens
llama_perf_context_print:    graphs reused =         48
llama_memory_breakdown_print: | memory breakdown [MiB] | total   free    self   model   context   compute    unaccounted |
llama_memory_breakdown_print: |   - Vulkan0 (RTX 2060) |  6144 = 4138 + (1458 =   934 +     224 +     299) +         547 |
llama_memory_breakdown_print: |   - Host               |                  136 =   125 +       0 +      11                |
Segmentation fault (core dumped)

CUDA

$ build/bin/llama-cli -m /opt/hf_models/qwen2.5-1.5b-instruct-q4_k_m.gguf -t 8 --no-mmap -fa on --cache-type-k f32 --cache-type-v f32 -ngl 99 -no-cnv -p "Write me a dog walking business idea 1. " -n 50 --seed 2362421855

Write me a dog walking business idea 1. 2. 3.

Sure, here are three dog walking business ideas for you:

1. **Doggy Digs**: This business concept revolves around providing dog walking services with a focus on outdoor adventures. You can offer a variety of activities such as

llama_perf_sampler_print:    sampling time =       3.85 ms /    61 runs   (    0.06 ms per token, 15835.93 tokens per second)
llama_perf_context_print:        load time =     509.04 ms
llama_perf_context_print: prompt eval time =      30.55 ms /    11 tokens (    2.78 ms per token,   360.02 tokens per second)
llama_perf_context_print:        eval time =     334.66 ms /    49 runs   (    6.83 ms per token,   146.42 tokens per second)
llama_perf_context_print:       total time =     392.07 ms /    60 tokens
llama_perf_context_print:    graphs reused =         48
llama_memory_breakdown_print: | memory breakdown [MiB] | total   free    self   model   context   compute    unaccounted |
llama_memory_breakdown_print: |   - CUDA0 (RTX 2060)   |  5737 = 4047 + (1458 =   934 +     224 +     299) +         231 |
llama_memory_breakdown_print: |   - Host               |                  136 =   125 +       0 +      11                |

Token Generation performance is similar between Vulkan and CUDA. @MaoJianwei can you test with Linux?

[JohannesGaessler]

Pascal and Turing are not directly comparable because the optimal code paths are very different.

[0cc4m]

For the Vulkan side, you will probably get better prompt processing performance if you use a legacy quant model (q*_0/q*_1), because then it can actually use the integer dot (dp4a) path. It is not yet implemented for other quants.

[MaoJianwei]

I cannot switch to linux environment @taronaeo

[MaoJianwei]

I think Q4_K_M also means integer dot, is that right? @0cc4m

Can it use the integer dot path?

[0cc4m]

No

[MaoJianwei]

change to use Q4_1, there is performance gap either. @0cc4m

Vulkan:
245.44 tokens per second for prefill phase
31.32 tokens per second for decode phase

CUDA:
502.46 tokens per second for prefill phase
14.11 tokens per second for decode phase

PS F:\llm\llama-b6568-bin-win-vulkan-x64> .\llama-server.exe --model ..\Qwen2.5-1.5B-Instruct-Q4_1.gguf  --no-mmap --jinja --verbose-prompt    --host 0.0.0.0  --flash-attn on --cache-type-k f32 --cache-type-v f32 -ngl 100 --metrics


main: server is listening on http://0.0.0.0:8080 - starting the main loop
srv  update_slots: all slots are idle
srv  params_from_: Chat format: Hermes 2 Pro
slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 2144
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 2048, n_tokens = 2048, progress = 0.955224
slot update_slots: id  0 | task 0 | kv cache rm [2048, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 2144, n_tokens = 96, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_past = 2144, n_tokens = 96
slot      release: id  0 | task 0 | stop processing: n_past = 2850, truncated = 0
slot print_timing: id  0 | task 0 |
prompt eval time =    8735.50 ms /  2144 tokens (    4.07 ms per token,   245.44 tokens per second)
       eval time =   22572.65 ms /   707 tokens (   31.93 ms per token,    31.32 tokens per second)
      total time =   31308.15 ms /  2851 tokens
srv  update_slots: all slots are idle

PS F:\llm\llama-b6568-bin-win-cuda-12.4-x64> .\llama-server.exe --model ..\Qwen2.5-1.5B-Instruct-Q4_1.gguf  --no-mmap --jinja --verbosggml_cuda_init: GGML_CUDA_FORCE_MMQ:    non --cache-type-k f32 --cache-type-v f32 -ngl 100 --metrics


main: server is listening on http://0.0.0.0:8080 - starting the main loop
srv  update_slots: all slots are idle
srv  params_from_: Chat format: Hermes 2 Pro
slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 2144
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 2048, n_tokens = 2048, progress = 0.955224
slot update_slots: id  0 | task 0 | kv cache rm [2048, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 2144, n_tokens = 96, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_past = 2144, n_tokens = 96
slot      release: id  0 | task 0 | stop processing: n_past = 2902, truncated = 0
slot print_timing: id  0 | task 0 |
prompt eval time =    4267.01 ms /  2144 tokens (    1.99 ms per token,   502.46 tokens per second)
       eval time =   53791.22 ms /   759 tokens (   70.87 ms per token,    14.11 tokens per second)
      total time =   58058.23 ms /  2903 tokens
srv  update_slots: all slots are idle