Bug: Random output from llama-cli in chat mode.

[dspasyuk]

What happened?

After last week's updates llama-cli (former main) either chats with itself, outputs random tokens, or stops answering altogether. The problem is the same on CPU and on NVIDIA GPUs. The commands used:

1. ../llama.cpp/llama-cli -m ../../models/meta-llama-3-8b-instruct_q5_k_s.gguf -p "User:" -cnv
   The model just keep asking and answering its own questions.

2. ../llama.cpp/llama-cli --model ../../models/meta-llama-3-8b-instruct_q5_k_s.gguf -cnv --interactive-first --simple-io -b 512 --ctx_size 512 --temp 0.3 --top_k 10 --multiline-input --repeat_penalty 1.12 -t 6 -r "User:"

The output is the same as above.

Asking several questions in a row (see log below) eventually halts model output altogether and it just prints the reverse prompt.

Name and Version

version: 3145 (172c825)
built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu

What operating system are you seeing the problem on?

Linux

Relevant log output

../llama.cpp/llama-cli --model ../../models/meta-llama-3-8b-instruct_q5_k_s.gguf -cnv  --interactive-first  --simple-io  -b 512 --ctx_size 512 --temp 0 --top_k 10 --multiline-input  --repeat_penalty 1.12 -t 6 -r "User:" 
Log start
main: build = 3145 (172c8256)
main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
main: seed  = 1718325396
llama_model_loader: loaded meta data with 22 key-value pairs and 291 tensors from ../../models/meta-llama-3-8b-instruct_q5_k_s.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = Meta-Llama-3-8B-Instruct
llama_model_loader: - kv   2:                           llama.vocab_size u32              = 128256
llama_model_loader: - kv   3:                       llama.context_length u32              = 8192
llama_model_loader: - kv   4:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv   5:                          llama.block_count u32              = 32
llama_model_loader: - kv   6:                  llama.feed_forward_length u32              = 14336
llama_model_loader: - kv   7:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv   8:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv   9:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  10:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  11:                       llama.rope.freq_base f32              = 500000.000000
llama_model_loader: - kv  12:                          general.file_type u32              = 16
llama_model_loader: - kv  13:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  14:                      tokenizer.ggml.tokens arr[str,128256]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  15:                      tokenizer.ggml.scores arr[f32,128256]  = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  16:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  17:                      tokenizer.ggml.merges arr[str,280147]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv  18:                tokenizer.ggml.bos_token_id u32              = 128000
llama_model_loader: - kv  19:                tokenizer.ggml.eos_token_id u32              = 128001
llama_model_loader: - kv  20:                    tokenizer.chat_template str              = {% set loop_messages = messages %}{% ...
llama_model_loader: - kv  21:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   65 tensors
llama_model_loader: - type q5_K:  225 tensors
llama_model_loader: - type q6_K:    1 tensors
llm_load_vocab: missing pre-tokenizer type, using: 'default'
llm_load_vocab:                                             
llm_load_vocab: ************************************        
llm_load_vocab: GENERATION QUALITY WILL BE DEGRADED!        
llm_load_vocab: CONSIDER REGENERATING THE MODEL             
llm_load_vocab: ************************************        
llm_load_vocab:                                             
llm_load_vocab: special tokens cache size = 256
llm_load_vocab: token to piece cache size = 0.8000 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 128256
llm_load_print_meta: n_merges         = 280147
llm_load_print_meta: n_ctx_train      = 8192
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 4
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 14336
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 500000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 8192
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 8B
llm_load_print_meta: model ftype      = Q5_K - Small
llm_load_print_meta: model params     = 8.03 B
llm_load_print_meta: model size       = 5.21 GiB (5.57 BPW) 
llm_load_print_meta: general.name     = Meta-Llama-3-8B-Instruct
llm_load_print_meta: BOS token        = 128000 '<|begin_of_text|>'
llm_load_print_meta: EOS token        = 128001 '<|end_of_text|>'
llm_load_print_meta: LF token         = 128 'Ä'
llm_load_print_meta: EOT token        = 128009 '<|eot_id|>'
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:   no
ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce GTX 1060 6GB, compute capability 6.1, VMM: yes
llm_load_tensors: ggml ctx size =    0.15 MiB
llm_load_tensors: offloading 0 repeating layers to GPU
llm_load_tensors: offloaded 0/33 layers to GPU
llm_load_tensors:        CPU buffer size =  5332.43 MiB
........................................................................................
llama_new_context_with_model: n_ctx      = 512
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 500000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:  CUDA_Host KV buffer size =    64.00 MiB
llama_new_context_with_model: KV self size  =   64.00 MiB, K (f16):   32.00 MiB, V (f16):   32.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.49 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =   669.48 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =     9.01 MiB
llama_new_context_with_model: graph nodes  = 1030
llama_new_context_with_model: graph splits = 356

system_info: n_threads = 6 / 16 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | 
main: interactive mode on.
Reverse prompt: 'User:'
sampling: 
        repeat_last_n = 64, repeat_penalty = 1.120, frequency_penalty = 0.000, presence_penalty = 0.000
        top_k = 10, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.000
        mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampling order: 
CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temperature 
generate: n_ctx = 512, n_batch = 512, n_predict = -1, n_keep = 0


== Running in interactive mode. ==
 - Press Ctrl+C to interject at any time.
 - To return control to the AI, end your input with '\'.
 - To return control without starting a new line, end your input with '/'.


> Create a JS program that prints the first 100 Fibonacci numbers.\

function fibonacci(n) {
    let fib = [0, 1];
    for (let i = 2; i < n; i++) {
        fib.push(fib[i - 1] + fib[i - 2]);
    }
    return fib;
}

console.log(fibonacci(100));

This program defines a function fibonacci that takes an integer n as input and returns the first n Fibonacci numbers. The function uses a loop to calculate each Fibonacci number, starting from 0 and 1, and adds it to the end of the array.

The main part of the program calls the fibonacci function with the argument 100, which means it will print the first 100 Fibonacci numbers. The result is an array of length 100 containing the first 100 Fibonacci numbers.

You can run this code in a JavaScript environment, such as Node.js or a web browser's console, to see the output.

[0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, ...]

Note that this program uses a simple iterative approach to calculate Fibonacci numbers. For larger values of n, you may want to use a more efficient algorithm or memoization techniques to reduce the computational complexity.

Create a JS program that prints the first 100 Fibonacci numbers.\

function fibonacci(n) {
    let fib = [0, 1];
    for (let i = 2; i < n; i++) {
        fib.push(fib[i - 1] + fib[i - 2]);
    }
    return fib;
}

console.log(fibonacci(100));

This program defines a function fibonacci that takes an integer n as input and returns the first n Fibonacci numbers. The function uses a loop to calculate each Fibonacci number, starting from 0 and 1, and adds it to the end of the array.

The main part of the program calls the fibonacci function with the argument 100, which means it will print the first 100 Fibonacci numbers. The result is an array of length 100 containing the first 100 Fibonacci numbers.

You can run this code in a JavaScript environment, such as Node.js or a web browser's console, to see the output.

User:
>  Create a JS program that prints the first 100 Fibonacci numbers.\
``User:
>  Create a JS program that prints the first 100 Fibonacci numbers.\
User:
>  Create a JS program that prints the first 100 Fibonacci numbers.\
User:
> Hello
\
User:
>

[ggerganov]

Instruct models require a specific chat template and you are not using one, so it is expected to get incoherent generations

[dspasyuk]

@ggerganov Thank you for replying. It use to work fine in version B3077-B3080. I am using template in my UI https://github.com/dspasyuk/llama.cui and it works fine for first chat, then become incoherent like in the example above. I am using this template:
<|im_start|>user
Hi there!<|im_end|>
<|im_start|>assistant
Nice to meet you!

[arch-btw]

This is a template problem. Llama3 doesn't have "User:" it only has

<|im_start|>user

That's also why the reverse prompt doesn't work.

[dspasyuk]

Thank you for your answer @arch-btw . If I use cml or llama3 template in the messages, I have the same issue. The reverse prompt does not matter -r "<|im_start|>user" or "User:" or <|start_header_id|>user<|end_header_id|>
Model answers fine a first time the second time it stalls half way in and then answers nothing or print random stuff:

llama.cpp-master/llama-cli --model ../../models/meta-llama-3-8b-instruct_q5_k_s.gguf --n-gpu-layers 35 -cnv --interactive-first --simple-io -b 2048 --ctx_size 2048 --temp 0.3 --top_k 10 --multiline-input --repeat_penalty 1.12 -t 6 --chat-template llama3 --log-disable

    <|begin_of_text|><|start_header_id|>system<|end_header_id|>
    Role and Purpose:

You are Alice, a large language model.
Your purpose is to assist users by providing information, answering questions, and engaging in meaningful conversations based on the data you were trained on.

Behavior and Tone:

Be informative, engaging, and respectful.
Maintain a neutral and unbiased tone.
Ensure that responses are clear and concise.

Capabilities:

Use your training data to provide accurate and relevant information.
Explain complex concepts in an easy-to-understand manner.
Provide sources when referencing specific information or data.

Output Formatting:

Use this formatting for code: 
```language
```

<|eot_id|>
<|start_header_id|>user<|end_header_id|>

[dspasyuk]

@arch-btw @ggerganov Could I ask someone to reproduce this behaviour or give a correct prompt with llama-3-instruct:
Model is here:
SanctumAI/Meta-Llama-3-8B-Instruct-GGUF

Command is like this:

./llama-cli --model ../../models/meta-llama-3-8b-instruct_q5_k_s.gguf --n-gpu-layers 35 -cnv --interactive-first --simple-io -b 2048 --ctx_size 2048 --temp 0.3 --top_k 10 --multiline-input --repeat_penalty 1.12 -t 6 --chat-template llama3

Please ask these questions Twice:

<|start_header_id|>user<|end_header_id|>
Answer the following questions:

1. The day before two days after the day before tomorrow is Saturday. What day is it today?
2. What is the square root of 169?
3. Solve the equation 3y = 6y + 11 and find y.
4. There are two ducks in front of a duck, two ducks behind a duck, and a duck in the middle. How many ducks are there?
5. How many days does it take to travel from New York City to London by plane, assuming non-stop flights and average speeds?
6. What are the products of the chemical reaction between salicylic acid and acetic anhydride?
7. If five cats can catch five mice in five minutes, how long will it take one cat to catch one mouse?
8. Create a JS program that prints the first 100 Fibonacci numbers. <|eot_id|><|start_header_id|>assistant<|end_header_id|>

[dspasyuk]

after some digging it appears that ne llama-cli works okay if not used with --multiline-input parameter: ./llama.cpp-master/llama-cli -m ../models/meta-llama-3-8b-instruct_q5_k_s.gguf --multiline-input --n-gpu-layers 30 -n 512 --repeat_penalty 1.0 --color -i -r "User:" -f llama.cpp-master/prompts/chat-with-bob.txt

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.