Eval bug: <think> tag with DeepSeek-R1-Distill-Qwen-1.5B-Q5_K_M.gguf

[gnusupport]

Name and Version

llama-cli --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce GTX 1050 Ti, compute capability 6.1, VMM: yes
version: 159 (80d0d6b)
built with Debian clang version 14.0.6 for x86_64-pc-linux-gnu

I see tag, which I should not see:

<think>
Okay, the user just sent "Hello". I should respond in a friendly and welcoming manner. Maybe say hello back and ask how I can assist them today. Keep it simple and open-ended so they can express what they need help with.
</think>

Hello! How can I assist you today?

with DeepSeek-R1-Distill-Qwen-1.5B-Q5_K_M.gguf

Operating systems

Linux

GGML backends

CUDA

Hardware

i5 + GTX 1050 Ti 4 GB

Models

DeepSeek-R1-Distill-Qwen-1.5B-Q5_K_M.gguf

Problem description & steps to reproduce

Okay, the user just sent "Hello". I should respond in a friendly and welcoming manner. Maybe say hello back and ask how I can assist them today. Keep it simple and open-ended so they can express what they need help with.

Hello! How can I assist you today?

First Bad Commit

No response

Relevant log output

none

[ngxson]

It's the main feature of R1 series, not a bug....

[Mushoz]

I am not sure what the right solution to this problem would be. Especially for the llama-server application instead of the cli. On one hand, you do want to be able to view the contents between the thinking tags for 2 reasons:

1. It's interesting to see how a model got to its answer, and it wouldn't be the first time it had the correct solution at one point in its train of thought but then settled on the wrong one. So the content in between the tags is definitely interesting to have access to.
2. If the thinking process is going off the rails, you are able to quit the generation midway when running it in streaming mode.

Having said that, Deepseek does recommend removing the content in between the thinking tags when dealing with multi-turn conversations as it blows up the context needlessly and makes it more difficult for the model to focus on the actual answer it ended up giving. If inference software leaves this content in, then clients will have to remove it to support proper multi-turn conversations with these models. That's a LOT of duplicate work for all the clients potentially using these models. On the other hand, removing them in the inference software prevents you from accessing this information at all.

Perhaps functionality to filter this out could be behind a llama-cli and llama-server switch?

[ngxson]

IMO you will have less performance by removing the think process from conversation. If you keep the old message as-is, llama.cpp will be able to reuse KV cache for these tokens and we only need to process only new tokens.

And if you still want to do:

• for server, it should be up to the client to remove it
• for llama-cli, almost impossible for now because it's designed to add messages, but not to go back and modify old message

[ngxson]

Also I should mention that support for jinja is on the horizon, and can help archive this by activating jinja engine

[Mushoz]

I am not sure what would perform best. On one hand as you mention it will invalidate the KV cache forcing prompt re-processing. On the other hand, token generation also slows down with increasing context. So keeping the context shorter (by removing the thinking content) will make the generation faster. So this might even out in the end depending on multiple variables.

Furthermore, the thinking contents are only removed from the most recent reply from the model since all previous replies would have already had their thinking content removed (And therefor have already been prompt processed in their altered form). From my understanding of how the KV cache is implemented (I might be wrong here!), in a multi turn conversation the cache would still be able to be used for all replies 1 through N-1. Only the last reply N will have to be reprocessed (as well as your own prompt of course) since that's the reply that will be edited by removing the thinking content. But that reply should be concise since the train of thought has been removed and be quick to process.

Having said that, the performance conversation can become pretty moot pretty quickly if it causes the model to have poor accuracy because the thinking content is retained for older replies in multi-turn conversations. Qwen models tent to completely derail with context shifts, and with how chatty these models are, hitting the 32k context limit is really not difficult at all in multi turn conversations. Ideally functionality to remove the thinking content is implemented in the client as you mentioned, but I am not sure how realistic that is going to be.

For people using Openweb UI, a solution has already been implemented thankfully: https://www.reddit.com/r/LocalLLaMA/comments/1i6b65q/better_r1_experience_in_open_webui/

I would also love to give these models a try in Aider and Cline, but right now they aren't useful due to this limitation.

By the way, what is this Jinja you mentioned? Can I read up on these plans somewhere? Curious to see how this fits in with these models.

[ngxson]

By the way, what is this Jinja you mentioned? Can I read up on these plans somewhere? Curious to see how this fits in with these models.

#11016

[gnusupport]

It's the main feature of R1 series, not a bug....

I appreciate the feature, I just think that representation should be different.

• I don't think user is supposed to see the tags
• I think that "thinking" progress could be somehow separate from the actual response
• Copy function of the actual response should copy response only
• There can be separate copy function for the "thinking" part
• "Thinking" part could be somehow collapsed and not necessarily visible

[ngxson]

What you're asking are "new features" and not "bugs".

[Mushoz]

I haven't been able to test this myself, but apparently the API responses from r1 have "content" and "reasoning_content" in the body that is returned. Maybe llama-server could implement something similar as to be compatible with clients that rely on this r1 specific format? Just thinking out loud here.

[ngxson]

@Mushoz yes it can be a good idea, you can make a PR or a dedicated issue if you like

[Mushoz]

I found the documentation that mentions most of the things discussed here: https://api-docs.deepseek.com/guides/reasoning_model

How should this content and reasoning_content separation work in streaming mode though? I have not used the DeepSeek API myself, so I am not sure how it makes this separation when the LLM's reply is streamed to the client instead of returning the entire reply at the end.

I haven't used this application myself yet, but using a proxy might also be a helpful stopgap solution for now: https://github.com/bold84/cot_proxy

[Mushoz]

@ngxson Should adding --jinja get rid of the content in the thinking tags? From llama-server's documentation, the default template should be the template taken from model's metadata.

Looking at the template here: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B/blob/main/tokenizer_config.json

I can see: {% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}

But setting the --jinja switch is not getting rid of the tags in the replies I am getting through openweb ui. Furthermore, using llama-server's builtin GUI I also still see the thinking tags. Am I misunderstanding how things should be working now?

[ngxson]

Should adding --jinja get rid of the content in the thinking tags?

I don't know, you can try and tell us.

The jinja template is stored inside gguf, should be picked up with --jinja, we just need to see if it's compatible with our minja engine or not.