Misc. bug: Qwen3 30B A3B Q4_K_M loads on server but quickly dies after requesting inference through Llama.cpp web UI

[sidran]

Name and Version

Version (release): B5215 Windows Vulkan x64

Operating systems

Windows

Which llama.cpp modules do you know to be affected?

No response

Command line

echo Running Qwen3 30B MoE server 12 layers 12288 context

llama-server.exe ^
--model "D:\LLMs\Qwen3-30B-A3B-Q4_K_M.gguf" ^
--gpu-layers 12 ^
--ctx-size 12288 ^
--samplers top_k;dry;min_p;temperature;typ_p;xtc ^
--top-k 40 ^
--dry-multiplier 0.5 ^
--min-p 0.00 ^
--temp 0.6 ^
--top-p 0.95 ^
--repeat-penalty 1.1

Problem description & steps to reproduce

Edit: GGUF was downloaded from ggml's HF repository
(https://huggingface.co/ggml-org/Qwen3-30B-A3B-GGUF/blob/main/Qwen3-30B-A3B-Q4_K_M.gguf)

It loads and seems everything is ok but as soon as I request inference through Llama.cpp's web UI, I get this error

First Bad Commit

No response

Relevant log output

[Thellton]

I'm getting the same issue. did a little digging and gave Google Gemini the method that the assertion is being thrown in, and based upon what I understood, I believe as a temporary measure, using the --batch-size flag to reduce the batch size from the standard 2048 tokens processed to 365 tokens or less for example will prevent the assertion occurring.

testing involved submitting 6644 tokens in a single go with various batch-size values ranging from 2048 to 512 which all failed with 365 and below working, as well as several small multi-turn conversations using small batch-sizes (16 to 64 tokens processed at any one time).

EDIT: I'm using IQ3_XS quant running on an Arc A770 16GB.

[sidran]

I'm getting the same issue. did a little digging and gave Google Gemini the method that the assertion is being thrown in, and based upon what I understood, I believe as a temporary measure, using the --batch-size flag to reduce the batch size from the standard 2048 tokens processed to 365 tokens or less for example will prevent the assertion occurring.

testing involved submitting 6644 tokens in a single go with various batch-size values ranging from 2048 to 512 which all failed with 365 and below working, as well as several small multi-turn conversations using small batch-sizes (16 to 64 tokens processed at any one time).

EDIT: I'm using IQ3_XS quant running on an Arc A770 16GB.

Thanks comrade!
I confirm that this actually works. Model seems awesome. I have 32Gb RAM and AMD 6600 8Gb, offloading 14/48 layers to VRAM, with 12288 context length and get ~10.7 t/s on this 30B model.
Life is good 😸

[stduhpf]

I can confirm. Dense Qwen3 models work fine, but the MoE crashes. I'm also using Vulkan, the crash still happens with no GPU offloading. (Edit: it doesn't happen on CPU only build)

Changing batch size is a valid workaround. It seems I can go up to a batch size of 384, but 385 crashes.

[ggerganov]

Seems Vulkan related. At least with Metal both 30B-A3B and 235B-A22B work without issues.

[TheNexter]

same issue

srv  params_from_: Chat format: Content-only
slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 16384, n_keep = 0, n_prompt_tokens = 414
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 414, n_tokens = 414, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_past = 414, n_tokens = 414
/app/ggml/src/ggml-vulkan/ggml-vulkan.cpp:5052: GGML_ASSERT(nei0 * nei1 <= 3072) failed

Command line start :

llama-server -m /Models/Qwen_Qwen3-30B-A3B-Q4_K_M.gguf --port 9999 --ctx-size 16384 --gpu-layers 12 --temp 0.6 --top-k 20 --top-p 0.95 --flash-attn

Configuration :

Ubuntu 24.04 Host
Docker container for the running part (llama-swap i remplace the binary server by : b5218)
GPU : Amd 6600XT 8Go
RAM : 46Go

If you need more info reply to my comment, ready to test some weird stuff if needed.

[Ayandaftary]

Same thing happens on my 6900XT but only with vulkan, ROCm works fine but vulkan is faster.

[stan4cb]

I've same problem, mostly at second prompt crash with
GGML_ASSERT(nei0 * nei1 <= 3072) failed
Qwen3-30B-A3B-Q4_K_M.gguf or Qwen3-30B-A3B-UD-Q4_K_XL.gguf doesn't matter
llama-server -m Qwen3-30B-A3B-UD-Q4_K_XL.gguf or -ngl 100 doesn't matter

--batch-size fixes

using b5220 Vulkan with 9070 XT

[DD3Boh]

Same thing happens on 780M APU, using Vulkan.

[mlsterpr0]

Isn't it similar to GLM-4-32b? which still only works if i use -b 8 -ub 8? Otherwise gibberish output... (vulkan version not cpu only)

[ZUIcat]

Same thing happens on 780M APU, using Vulkan, +1

[Dominguero]

Same with Vega56.

Another thing I've noticed, which I don't know if it has anything to do with it, is that the Vulkan version works 10%-20% slower than the AVX2 version.

This is something I've only seen with this model, and the other Qwen 225b MOE.

[mrdevolver]

My HW Specs:
OS: Windows 10 64bit
CPU: AMD Ryzen 7 2700X (8c / 16t)
RAM: 16 GB
GPU: Radeon RX Vega 56
VRAM: 8 GB

Tested model: Unsloth/Qwen3-30B-A3B-Q3_K_M.gguf

I have the same problem with Vulkan version. (I haven't tried CPU version yet).

Couple of notes from my own observation, hopefully it will be of any use for developers for fixing this issue:

• LM Studio, text-generation-webui, KoboldCpp (all based on llama.cpp). Model works in the UI of the LM Studio itself (haven't tested UI usage in the rest), but crashes in all of these apps as soon as I try to call it through the llama.cpp OpenAI API server they provide (calls are done by third party UIs).
• In LM Studio UI I can use batch size even 4096 and it still works.
• Perhaps the most interesting: LM Studio gives quick cURL command for testing the API server. Interestingly, when I called the same API server through that test cURL command, it did NOT crash and gave response as expected.

Little anecdote from testing: cURL command for testing API as given by LM Studio is not ready for use in Windows command line neither Windows batch file (.bat), so while I still had Qwen3-30B-A3B loaded in the LM Studio, I asked the model in the UI itself to change the cURL command to make it work in Windows batch script. It did a perfect job, so then I was able to test the API simply using cURL thanks to which I found out it works that way. 😂

[Marcuss2]

Running with llama-server --model Qwen3-30B-A3B-UD-Q4_K_XL.gguf --n-gpu-layers 99 -ot ".ffn_.*_exps.=CPU" --ctx-size 32772 --flash-attn

It does happen to me as well. Latest build 5233
GPU: RX 6800

ROCm and CPU only backends don't do that, --flash-attn has no effect on the bug.