Misc. bug: llama-server builds possibly erroneous prompt for gemma 3

[mfritz2008]

Name and Version

b5621, built with cc (Debian 14.2.0-19) 14.2.0 for x86_64-linux-gnu

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

./llama-server -m google_gemma-3-27b-it-Q8_0.gguf -b 256 -ub 256 -fa -n -1 --host 0.0.0.0 --port 8001 --verbose --verbose-prompt --log-timestamps --slot-save-path ./

Problem description & steps to reproduce

When using gemma-3-27b-it, the prompt built by llama.cpp from a message array have two issues:

1. Assistant message appears before system message;
2. When there are multiple system messages, only the last one is included in the prompt.

How to test:

curl -X POST --location "http://localhost:8001/apply-template" \
    -H "Content-Type: application/json" \
    -d '{
          "messages": [
            {
              "role": "system",
              "content": "System message 1"
            },
            {
              "role": "system",
              "content": "System message 2"
            },
            {
              "role": "assistant",
              "content": "I am your assistant."
            },
            {
              "role": "user",
              "content": "Hello!"
            },
            {
              "role": "assistant",
              "content": "How can I help you?"
            },
            {
              "role": "user",
              "content": "Tell me a story."
            }
          ]
        }'

Actual result:

{"prompt":"<start_of_turn>model\nI am your assistant.<end_of_turn>\n<start_of_turn>user\nSystem message 2\n\nHello!<end_of_turn>\n<start_of_turn>model\nHow can I help you?<end_of_turn>\n<start_of_turn>user\nTell me a story.<end_of_turn>\n<start_of_turn>model\n"}

The prompt have the first assistant message at the front, and only System message 2 is included.

Expected result:

• Built prompt should have the messages appear in the above order, assistant message should not appear in the front.
• All system messages should be included.

---

However, when using other models, such as Mistral Nemo (Mistral V3), the prompt is built as expected, with all the parts, and in correct order:

{"prompt":"[INST] System message 1\nSystem message 2\nI am your assistant.</s>[INST] Hello! [/INST]How can I help you?</s>[INST] Tell me a story. [/INST]"}

... and Mistral Small 2501 (Mistral V7), the prompt is built as expected:

{"prompt":"[SYSTEM_PROMPT]System message 1[/SYSTEM_PROMPT][SYSTEM_PROMPT]System message 2[/SYSTEM_PROMPT]I am your assistant.</s>[INST]Hello![/INST]How can I help you?</s>[INST]Tell me a story.[/INST]"}

... and also Qwen2.5 (ChatML), the prompt is built as expected, with all the parts, and in correct order:

{"prompt":"<|im_start|>system\nSystem message 1<|im_end|>\n<|im_start|>system\nSystem message 2<|im_end|>\n<|im_start|>assistant\nI am your assistant.<|im_end|>\n<|im_start|>user\nHello!<|im_end|>\n<|im_start|>assistant\nHow can I help you?<|im_end|>\n<|im_start|>user\nTell me a story.<|im_end|>\n<|im_start|>assistant\n"}

[mfritz2008]

I'm using a simple workaround as shown below. It produces the desired result, though I'm not sure about correctness.

------------------------------ src/llama-chat.cpp ------------------------------
index d12743e6..c9656ceb 100644
@@ -325,7 +325,26 @@ int32_t llm_chat_apply_template(
             ss << "<s>assistant\n";
         }
     } else if (tmpl == LLM_CHAT_TEMPLATE_GEMMA) {
+        for (auto message : chat) {
+            std::string role(message->role);
+            if (role == "system") {
+                // there is no system message for gemma, but we will send it as user prompt, so nothing is broken
+                role = "user";
+            } else if (role == "assistant") {
+                // in gemma, "assistant" is "model"
+                role = "model";
+            }
+            ss << "<start_of_turn>" << role << "\n" << message->content << "<end_of_turn>\n";
+        }
+        if (add_ass) {
+            ss << "<start_of_turn>model\n";
+        }
+        
         // google/gemma-7b-it
+        /*
         std::string system_prompt = "";
         for (auto message : chat) {
             std::string role(message->role);
@@ -346,6 +365,7 @@ int32_t llm_chat_apply_template(
         if (add_ass) {
             ss << "<start_of_turn>model\n";
         }
+        */
     } else if (tmpl == LLM_CHAT_TEMPLATE_ORION) {
         // OrionStarAI/Orion-14B-Chat
         std::string system_prompt = "";

The test returns seemingly correct prompt:

{"prompt":"<start_of_turn>user\nSystem message 1<end_of_turn>\n<start_of_turn>user\nSystem message 2<end_of_turn>\n<start_of_turn>model\nI am your assistant.<end_of_turn>\n<start_of_turn>user\nHello!<end_of_turn>\n<start_of_turn>model\nHow can I help you?<end_of_turn>\n<start_of_turn>user\nTell me a story.<end_of_turn>\n<start_of_turn>model\n"}

[ggerganov]

1. Assistant message appears before system message;

I think this is intentional - see the comment in the source code:

llama.cpp/src/llama-chat.cpp

Lines 330 to 337 in a681b4b

	for (auto message : chat) {
	std::string role(message->role);
	if (role == "system") {
	// there is no system message for gemma, but we will merge it with user prompt, so nothing is broken
	system_prompt = trim(message->content);
	continue;
	}
	// in gemma, "assistant" is "model"

[mfritz2008]

1. Assistant message appears before system message;

I think this is intentional - see the comment in the source code:

// there is no system message for gemma, but we will merge it with user prompt, so nothing is broken

@ggerganov Yes... but this does not work too well in practice.

The format in the main post (system - ... - system - assistant - user - ...) is widely used in popular frontends such as SillyTavern, where various components of a character card (worldInfo, lore, character description, personality...) is sent as system role messages, and there is a "greeting" assistant message before the actual interaction begins.

So my proposal is:

1. Do not overwrite previous system messages with later one, which is caused by system_prompt = trim(message->content); (later system messages should be appended, not assigned)
2. Do not merge system message to a later user message, instead just build them into the final prompt as separate user messages. (The current merge method messes with order if an assistant message (the greeting) comes right after system message, but before user messages. Since the bulk of the messages are system role(several thousand tokens of worldInfo, lore, character description...), putting the short greeting before the bulk completely defeats the purpose of the greeting.)

[ggerganov]

This sounds reasonable - feel free to open a PR.