Eval bug: Heavy nondeterminism in Qwen3 MoE (CUDA)

[matteoserva]

Name and Version

llama-cli --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
Device 0: NVIDIA GeForce RTX 4060 Ti, compute capability 8.9, VMM: yes
Device 1: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes
version: 5269 (1d36b36)
built with cc (Debian 12.2.0-14) 12.2.0 for x86_64-linux-gnu

Operating systems

Linux

GGML backends

CUDA

Hardware

rtx 5060ti 16GB + rtx4060ti 16GB

Models

Qwen_Qwen3-30B-A3B-Q6_K.gguf by bartowski.
sha256sum: d511d02955714b08ff1b4354d6eae8ea513179a83fa5498466db2731528074dd

Problem description & steps to reproduce

I'm using a grammar to simulate the nothink qwen prompt format. Sometimes the output is generated correctly, sometimes the model outputs the wrong token while still aligned with the grammar.

The command I'm using to test:

curl http://localhost:8080/completion -H "Content-Type: application/json" -d '{
  "prompt": "<|im_start|>system\n<|im_end|>\n<|im_start|>user\nhi<|im_end|>\n<|im_start|>assistant\n",
  "grammar": "root ::= \"<think>\\n\\n</think>\\n\\n\" .*",
  "temperature": 0.001,
  "n_predict": 6,
  "seed": 42
}'

Correct output: [151667 198 198 151668 ...] <think>\n\n</think>...
Wrong output: [151667 198 198 27 14 ...] <think>\n\n</...

Sometimes the model outputs the correct output, sometimes it outputs the wrong output and the following output breaks since the model cannot see the </think> token. I'm not restarting llama-server between tests and not changing the seed. I expect the model to always output the token 151668

Command line used to launch llama-server: /llama-server -ngl 175 -t 6 -c 32768 --host 0.0.0.0 -fa -ctk q8_0 -ctv q8_0 --slots -a current --temp 0.6

First Bad Commit

No response

Relevant log output

`{"index":0,"content":"<think>\n\n</think"...`

[CISC]

You can't use grammar to force a specific token, only a specific sequence of characters, if the model can complete that sequence with alternate tokens that's perfectly acceptable to the grammar.

[CISC]

If you are just trying to disable thinking you yourself have already implemented the means to do that, see #13212 (comment) :)

[matteoserva]

This is a different problem. The output shouldn't be random when the seed is fixed and the temperature is zero.

If I let the model generate its output "naturally", the generated tokens are always the same at every run.

If I use the grammar to force the exact same output string, then the output tokens can change randomly, producing extremely low probability tokens.

[CISC]

This is a different problem. The output shouldn't be random when the seed is fixed and the temperature is zero.

Shouldn't, but is, at least with CUDA unfortunately.

If I let the model generate its output "naturally", the generated tokens are always the same at every run.

If I use the grammar to force the exact same output string, then the output tokens can change randomly, producing extremely low probability tokens.

Well, grammar will naturally skew the probabilities, I think you're just seeing the results of the numerical instability of CUDA more frequently due to the limited token selection.

[CISC]

I'll reopen the issue in case someone wants to take a closer look, but you might want to change the title to a more accurate description of the issue.

[matteoserva]

I made some tests and I can provide more context. I used the following command to print the probabilities:

curl http://localhost:8080/completion -H "Content-Type: application/json" -d '{
  "prompt": "<|im_start|>system\n<|im_end|>\n<|im_start|>user\nhi<|im_end|>\n<|im_start|>assistant\n",
  "grammar": "root ::= \"<think>\\n\\n</think>\\n\\n\" .*",
  "temperature": 0.001,
  "n_predict": 4,
  "seed": 42,
  "n_probs": 4
}'

The 4 output tokens are

1. <think> \n \n </think> when the output is generated correctly
2. <think> \n \n < when the generation is wrong. The model then goes out of distribution and generates low quality output.

The logprob of the 4th token is:

1. </think> : -22.50234603881836,
2. <: -23.995466232299805

Sometimes I get the wrong result 30 times in a row, sometimes I get the correct result 30 times in a row. With these settings it should be less than 1 in a billion chance ( 2^30) of being correct behavior.

EDIT: nevermind. I have to test with "cache_prompt": false

[matteoserva]

Ok. I am able to narrow down the problem even more with this python script:

import requests
import json

headers = {
    'Content-Type': 'application/json',
}

json_data = {
    'prompt': '<|im_start|>system\n<|im_end|>\n<|im_start|>user\nhi<|im_end|>\n<|im_start|>assistant\n',
    'grammar': 'root ::= "<think>\\n\\n</think>\\n\\n" .*',
    'temperature': 0.001,
    'n_predict': 4,
    'seed': 42,
    'n_probs': 1,
    'cache_prompt': False,
}

for _ in range(10):
    response = requests.post('http://localhost:8080/completion', headers=headers, json=json_data)
    result = json.loads(response.text)
    fourth = result["completion_probabilities"][3]
    del fourth["top_logprobs"]
    del fourth["bytes"]
    print(fourth)

This is the output of Qwen3-MoE for CPU debug:

$ python3 qwen-mode-bug.py 
{'id': 27, 'token': '<', 'logprob': -21.9802303314209}
{'id': 27, 'token': '<', 'logprob': -21.9802303314209}
{'id': 27, 'token': '<', 'logprob': -21.9802303314209}
{'id': 27, 'token': '<', 'logprob': -21.9802303314209}
{'id': 27, 'token': '<', 'logprob': -21.9802303314209}
...

This is the output of Qwen3-MoE for CUDA Release:

$ python3 qwen-mode-bug.py 
{'id': 27, 'token': '<', 'logprob': -24.497241973876953}
{'id': 27, 'token': '<', 'logprob': -25.099708557128906}
{'id': 151668, 'token': '</think>', 'logprob': -22.38550567626953}
{'id': 27, 'token': '<', 'logprob': -22.939809799194336}
{'id': 27, 'token': '<', 'logprob': -23.012990951538086}
{'id': 27, 'token': '<', 'logprob': -24.535411834716797}
{'id': 151668, 'token': '</think>', 'logprob': -26.998727798461914}
{'id': 27, 'token': '<', 'logprob': -25.120460510253906}
{'id': 27, 'token': '<', 'logprob': -26.36814308166504}
{'id': 27, 'token': '<', 'logprob': -25.176483154296875}

This is the output with Qwen3-0.6 (Dense), CUDA Release:

{'id': 27, 'token': '<', 'logprob': -10.674346923828125}
{'id': 27, 'token': '<', 'logprob': -10.674346923828125}
{'id': 27, 'token': '<', 'logprob': -10.674346923828125}
{'id': 27, 'token': '<', 'logprob': -10.674346923828125}
...

problem

The log probabilities in Qwen-MoE (CUDA) vary wildly between -22 and -27, which is not expected. The inputs are always the same and the probability distribution should be always the same.
I might be wrong but I'm not expecting this behaviour.
@JohannesGaessler Maybe you can confirm this?

[JohannesGaessler]

Given the server parameters the output should to my knowledge be deterministic, even with CUDA. I have recently made quite a lot of changes to the MoE code. Unfortunately, as was pointed out to me in ggml-org/ggml#1178 (comment) , prior to my changes the results were already non-deterministic so I think a git bisect will be inconclusive.

Just to rule out the possibility of a hardware issue: do you get deterministic results after running nvidia-smi --lock-gpu-clocks 0,1000 --mode 1? I have a machine with multiple 4090s running off of a single 2050 W PSU and as it turns out even with just 3 of them running in parallel the power spikes can overlap in unlucky ways that either cause bit flips or system crashes.

[matteoserva]

Just to rule out the possibility of a hardware issue: do you get deterministic results after running nvidia-smi --lock-gpu-clocks 0,1000 --mode 1? I have a machine with multiple 4090s running off of a single 2050 W PSU and as it turns out even with just 3 of them running in parallel the power spikes can overlap in unlucky ways that either cause bit flips or system crashes.

It doesn't seem to change anything

Qwen-moe with default settings:

python3 qwen-mode-bug.py 
{'id': 151668, 'token': '</think>', 'logprob': -25.749713897705078}
{'id': 27, 'token': '<', 'logprob': -25.500146865844727}
{'id': 27, 'token': '<', 'logprob': -23.861719131469727}
{'id': 27, 'token': '<', 'logprob': -24.062076568603516}
{'id': 27, 'token': '<', 'logprob': -24.572689056396484}
{'id': 27, 'token': '<', 'logprob': -23.132158279418945}
{'id': 151668, 'token': '</think>', 'logprob': -22.906593322753906}
{'id': 27, 'token': '<', 'logprob': -23.740596771240234}
{'id': 27, 'token': '<', 'logprob': -23.677282333374023}
{'id': 27, 'token': '<', 'logprob': -24.058055877685547}

Qwen-moe with nvidia-smi --lock-gpu-clocks 0,1000 --mode 1

$ python3 qwen-mode-bug.py 
{'id': 27, 'token': '<', 'logprob': -24.385953903198242}
{'id': 27, 'token': '<', 'logprob': -24.010208129882812}
{'id': 151668, 'token': '</think>', 'logprob': -23.4265079498291}
{'id': 151668, 'token': '</think>', 'logprob': -25.16918182373047}
{'id': 151668, 'token': '</think>', 'logprob': -26.014835357666016}
{'id': 151668, 'token': '</think>', 'logprob': -24.031843185424805}
{'id': 27, 'token': '<', 'logprob': -23.161317825317383}
{'id': 151668, 'token': '</think>', 'logprob': -24.846834182739258}
{'id': 151668, 'token': '</think>', 'logprob': -23.85172462463379}
{'id': 27, 'token': '<', 'logprob': -24.06478500366211}

Qwen 32b (dense) with nvidia-smi --lock-gpu-clocks 0,1000 --mode 1

$ python3 qwen-mode-bug.py 
{'id': 151668, 'token': '</think>', 'logprob': -8.822237014770508}
{'id': 151668, 'token': '</think>', 'logprob': -8.822237014770508}
{'id': 151668, 'token': '</think>', 'logprob': -8.822237014770508}
{'id': 151668, 'token': '</think>', 'logprob': -8.822237014770508}
{'id': 151668, 'token': '</think>', 'logprob': -8.822237014770508}
[...]

[JohannesGaessler]

Please check this fix: #13294 . There was a race condition in the new MMQ code that could be causing this.