Using the latest llama-cli will result in an output that keeps blanking

[QIANXUNZDL123]

What happened?

I enter： who is Elon Mask? and after replying to a paragraph, the cursor starts to move all the way down. If you don't stop him, he'll keep going down

Name and Version

./bin/llama-cli -m ../modelGGUF/llama-2-7b-chat.Q4_0.gguf -cnv -t 6 --ctx_size 2048 -n -1 -b 256 --top_k 1000 --temp 0.2 --repeat_penalty 1.1

What operating system are you seeing the problem on?

No response

Relevant log output

No response

[ngxson]

You're using an old model that does not have chat template support. Add --chat-template llama2 to your command

[dspasyuk]

@QIANXUNZDL123 Switch to llama3-instruct QuantFactory/Meta-Llama-3-8B-Instruct-GGUF, llama2 is ancient, and use template like @ngxson has suggested also for prompt you will need to use: <|start_header_id|>user for reverse prompt check out llama.cui for more details https://github.com/dspasyuk/llama.cui

[ngxson]

@dspasyuk llama 3 is new model, so llama.cpp has auto chat template support for it.

Simply run llama 3 with ./llama-cli -m ... -cnv, no need to worry about special tokens

[dspasyuk]

@ngxson Yes, I know this is how it was intended to work but if you stress test it with this setting model starts to goof out after about 3000 tokens, at least in my hands:

../llama.cpp/llama-cli --model ../../models/Meta-Llama-3-8B-Instruct_Q5_K_S.gguf --n-gpu-layers 30 -cnv --interactive --interactive-first --simple-io -b 2048 -n -1 -e --ctx_size 0 --temp 0.5 --top_k 10 --multiline-input --repeat_penalty 1.12 --chat-template llama3 -t 8 -ptc 10 -r "user:"

Using llama3 template manually work without issues for infinity. I am not sure why there is a difference perhaps the issue is with missing BOS token in the template because llama-cli prints this as a template, which does not have BOS token:

main: chat template example: <|start_header_id|>system<|end_header_id|>

You are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>

Hello<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Hi there<|eot_id|><|start_header_id|>user<|end_header_id|>

How are you?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3/

[ngxson]

As far as I can tell, BOS is added in conversation mode. main.log says:

[1719953638] add_bos: 1
[1719953638] tokenize the prompt
[1719953638] prompt: "<|start_header_id|>system<|end_header_id|>

You are AI<|eot_id|>"
[1719953638] tokens: [ '<|begin_of_text|>':128000, '<|start_header_id|>':128006, 'system':9125, '<|end_header_id|>':128007, '':271, 'You':2675, ' are':527, ' AI':15592, '<|eot_id|>':128009 ]

Probably there is something to do with your sampling settings.

[dspasyuk]

Yes, anything is possible, could not figure it out, so switched to do it manually instead.

[dspasyuk]

@ngxson I do not think the reverse prompt is modified when --chat-template llama3 is used so using something like this User: is no good.

[ngxson]

@dspasyuk Conversation mode doesn't use reverse prompt. What I recommend is: you should either look into main.cpp code or check main.log file. When debugging something, it's best to have logs ; we don't just assuming how things works.

[dspasyuk]

@ngxson Llama-cli sets the reverse prompt in conversation mode in my logs, as for the manuals it says here.... see below, or do I missunderstand that?

Reverse Prompts
Reverse prompts are a powerful way to create a chat-like experience with a LLaMA model by pausing the text generation when specific text strings are encountered:

-r PROMPT, --reverse-prompt PROMPT: Specify one or multiple reverse prompts to pause text generation and switch to interactive mode. For example, -r "User:" can be used to jump back into the conversation whenever it's the user's turn to speak. This helps create a more interactive and conversational experience. However, the reverse prompt doesn't work when it ends with a space.

[ngxson]

Reverse prompts / insuffix / inprefix are used before there is chat template support. They are a different system.

Can you just post your main.log here? Without log data I can't see the bug.

[dspasyuk]

@ngxson Sure no problem here are two logs one with the template applied using --chat-template the other is manually formating prompt:
` Using Llama3 template

../llama.cpp/llama-cli --model ../../models/Meta-Llama-3-8B-Instruct_Q5_K_S.gguf --n-gpu-layers 25 -cnv --interactive --interactive-first --simple-io -b 2048 -n -1 -e --ctx_size 0 --temp 0.5 --top_k 10 --multiline-input --repeat_penalty 1.12 -t 8 -ptc 10 --chat-template llama3 -r "<|start_header_id|>user" --logdir ./ --log-append --no-display-prompt

[1719977797] Log start
[1719977797] Cmd: ../llama.cpp/llama-cli --model ../../models/Meta-Llama-3-8B-Instruct_Q5_K_S.gguf --n-gpu-layers 25 -cnv --interactive --interactive-first --simple-io -b 2048 -n -1 -e --ctx_size 0 --temp 0.5 --top_k 10 --multiline-input --repeat_penalty 1.12 -t 8 -ptc 10 --chat-template llama3 -r <|start_header_id|>user --logdir ./ --log-append --no-display-prompt
[1719977797] main: build = 0 (unknown)
[1719977797] main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
[1719977797] main: seed = 1719977797
[1719977797] main: llama backend init
[1719977797] main: load the model and apply lora adapter, if any
[1719977797] llama_model_loader: loaded meta data with 22 key-value pairs and 291 tensors from ../../models/Meta-Llama-3-8B-Instruct_Q5_K_S.gguf (version GGUF V3 (latest))
[1719977797] llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
[1719977797] llama_model_loader: - kv 0: general.architecture str = llama
[1719977797] llama_model_loader: - kv 1: general.name str = models
[1719977797] llama_model_loader: - kv 2: llama.block_count u32 = 32
[1719977797] llama_model_loader: - kv 3: llama.context_length u32 = 8192
[1719977797] llama_model_loader: - kv 4: llama.embedding_length u32 = 4096
[1719977797] llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336
[1719977797] llama_model_loader: - kv 6: llama.attention.head_count u32 = 32
[1719977797] llama_model_loader: - kv 7: llama.attention.head_count_kv u32 = 8
[1719977797] llama_model_loader: - kv 8: llama.rope.freq_base f32 = 500000.000000
[1719977797] llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
[1719977797] llama_model_loader: - kv 10: general.file_type u32 = 16
[1719977797] llama_model_loader: - kv 11: llama.vocab_size u32 = 128256
[1719977797] llama_model_loader: - kv 12: llama.rope.dimension_count u32 = 128
[1719977797] llama_model_loader: - kv 13: tokenizer.ggml.model str = gpt2
[1719977797] llama_model_loader: - kv 14: tokenizer.ggml.pre str = llama-bpe
[1719977797] llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,128256] = ["!", """, "#", "$", "%", "&", "'", ...
[1719977797] llama_model_loader: - kv 16: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
[1719977797] llama_model_loader: - kv 17: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
[1719977797] llama_model_loader: - kv 18: tokenizer.ggml.bos_token_id u32 = 128000
[1719977797] llama_model_loader: - kv 19: tokenizer.ggml.eos_token_id u32 = 128001
[1719977797] llama_model_loader: - kv 20: tokenizer.chat_template str = {% set loop_messages = messages %}{% ...
[1719977797] llama_model_loader: - kv 21: general.quantization_version u32 = 2
[1719977797] llama_model_loader: - type f32: 65 tensors
[1719977797] llama_model_loader: - type q5_K: 225 tensors
[1719977797] llama_model_loader: - type q6_K: 1 tensors
[1719977798] llm_load_vocab: special tokens cache size = 256
[1719977798] llm_load_vocab: token to piece cache size = 0.8000 MB
[1719977798] llm_load_print_meta: format = GGUF V3 (latest)
[1719977798] llm_load_print_meta: arch = llama
[1719977798] llm_load_print_meta: vocab type = BPE
[1719977798] llm_load_print_meta: n_vocab = 128256
[1719977798] llm_load_print_meta: n_merges = 280147
[1719977798] llm_load_print_meta: n_ctx_train = 8192
[1719977798] llm_load_print_meta: n_embd = 4096
[1719977798] llm_load_print_meta: n_head = 32
[1719977798] llm_load_print_meta: n_head_kv = 8
[1719977798] llm_load_print_meta: n_layer = 32
[1719977798] llm_load_print_meta: n_rot = 128
[1719977798] llm_load_print_meta: n_swa = 0
[1719977798] llm_load_print_meta: n_embd_head_k = 128
[1719977798] llm_load_print_meta: n_embd_head_v = 128
[1719977798] llm_load_print_meta: n_gqa = 4
[1719977798] llm_load_print_meta: n_embd_k_gqa = 1024
[1719977798] llm_load_print_meta: n_embd_v_gqa = 1024
[1719977798] llm_load_print_meta: f_norm_eps = 0.0e+00
[1719977798] llm_load_print_meta: f_norm_rms_eps = 1.0e-05
[1719977798] llm_load_print_meta: f_clamp_kqv = 0.0e+00
[1719977798] llm_load_print_meta: f_max_alibi_bias = 0.0e+00
[1719977798] llm_load_print_meta: f_logit_scale = 0.0e+00
[1719977798] llm_load_print_meta: n_ff = 14336
[1719977798] llm_load_print_meta: n_expert = 0
[1719977798] llm_load_print_meta: n_expert_used = 0
[1719977798] llm_load_print_meta: causal attn = 1
[1719977798] llm_load_print_meta: pooling type = 0
[1719977798] llm_load_print_meta: rope type = 0
[1719977798] llm_load_print_meta: rope scaling = linear
[1719977798] llm_load_print_meta: freq_base_train = 500000.0
[1719977798] llm_load_print_meta: freq_scale_train = 1
[1719977798] llm_load_print_meta: n_ctx_orig_yarn = 8192
[1719977798] llm_load_print_meta: rope_finetuned = unknown
[1719977798] llm_load_print_meta: ssm_d_conv = 0
[1719977798] llm_load_print_meta: ssm_d_inner = 0
[1719977798] llm_load_print_meta: ssm_d_state = 0
[1719977798] llm_load_print_meta: ssm_dt_rank = 0
[1719977798] llm_load_print_meta: model type = 8B
[1719977798] llm_load_print_meta: model ftype = Q5_K - Small
[1719977798] llm_load_print_meta: model params = 8.03 B
[1719977798] llm_load_print_meta: model size = 5.21 GiB (5.57 BPW)
[1719977798] llm_load_print_meta: general.name = models
[1719977798] llm_load_print_meta: BOS token = 128000 '<|begin_of_text|>'
[1719977798] llm_load_print_meta: EOS token = 128001 '<|end_of_text|>'
[1719977798] llm_load_print_meta: LF token = 128 'Ä'
[1719977798] llm_load_print_meta: EOT token = 128009 '<|eot_id|>'
[1719977798] llm_load_print_meta: max token length = 256
[1719977798] ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
[1719977798] ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
[1719977798] ggml_cuda_init: found 1 CUDA devices:
[1719977798] Device 0: NVIDIA GeForce GTX 1060 6GB, compute capability 6.1, VMM: yes
[1719977798] llm_load_tensors: ggml ctx size = 0.27 MiB
[1719977798] llm_load_tensors: offloading 25 repeating layers to GPU
[1719977798] llm_load_tensors: offloaded 25/33 layers to GPU
[1719977798] llm_load_tensors: CPU buffer size = 5332.43 MiB
[1719977798] llm_load_tensors: CUDA0 buffer size = 3575.78 MiB
[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977798] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799] .[1719977799]
[1719977799] llama_new_context_with_model: n_ctx = 8192
[1719977799] llama_new_context_with_model: n_batch = 2048
[1719977799] llama_new_context_with_model: n_ubatch = 512
[1719977799] llama_new_context_with_model: flash_attn = 0
[1719977799] llama_new_context_with_model: freq_base = 500000.0
[1719977799] llama_new_context_with_model: freq_scale = 1
[1719977799] llama_kv_cache_init: CUDA_Host KV buffer size = 224.00 MiB
[1719977799] llama_kv_cache_init: CUDA0 KV buffer size = 800.00 MiB
[1719977799] llama_new_context_with_model: KV self size = 1024.00 MiB, K (f16): 512.00 MiB, V (f16): 512.00 MiB
[1719977799] llama_new_context_with_model: CUDA_Host output buffer size = 0.49 MiB
[1719977799] llama_new_context_with_model: CUDA0 compute buffer size = 669.48 MiB
[1719977799] llama_new_context_with_model: CUDA_Host compute buffer size = 24.01 MiB
[1719977799] llama_new_context_with_model: graph nodes = 1030
[1719977799] llama_new_context_with_model: graph splits = 81
[1719977799] warming up the model with an empty run
[1719977799] n_ctx: 8192
[1719977799] main: chat template example: <|start_header_id|>system<|end_header_id|>

You are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>

Hello<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Hi there<|eot_id|><|start_header_id|>user<|end_header_id|>

How are you?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

[1719977799]
[1719977799] system_info: n_threads = 8 / 16 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
[1719977799] add_bos: 1
[1719977799] tokenize the prompt
[1719977799] prompt: "<|start_header_id|>system<|end_header_id|>

<|eot_id|>"
[1719977799] tokens: [ '<|begin_of_text|>':128000, '<|start_header_id|>':128006, 'system':9125, '<|end_header_id|>':128007, '':271, '<|eot_id|>':128009 ]
[1719977799] recalculate the cached logits (check): embd_inp.empty() false, n_matching_session_tokens 0, embd_inp.size() 6, session_tokens.size() 0, embd_inp.size() 6
[1719977799] main: interactive mode on.
[1719977799] Reverse prompt: '<|start_header_id|>user'
[1719977799] sampling:
repeat_last_n = 64, repeat_penalty = 1.120, frequency_penalty = 0.000, presence_penalty = 0.000
top_k = 10, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.500
mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
[1719977799] sampling order:
CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temperature
[1719977799] generate: n_ctx = 8192, n_batch = 2048, n_predict = -1, n_keep = 1
[1719977799]

[1719977799] == Running in interactive mode. ==
[1719977799] - Press Ctrl+C to interject at any time.
[1719977799] - To return control to the AI, end your input with ''.

• To return control without starting a new line, end your input with '/'.

[1719977799] embd_inp.size(): 6, n_consumed: 0
[1719977799] found an EOG token
[1719977799] eval: [ '<|begin_of_text|>':128000, '<|start_header_id|>':128006, 'system':9125, '<|end_header_id|>':128007, '':271, '<|eot_id|>':128009 ]
[1719977799] n_past = 6
[1719977799] embd_inp.size(): 8, n_consumed: 6
[1719977799] found antiprompt: !!!!!!!!!!!!!!!!!!!!!!!!<|begin_of_text|><|start_header_id|>system<|end_header_id|>

<|eot_id|><|start_header_id|>user
[1719977799] waiting for user input
[1719977808] buffer: '
Answer the following questions:

1. The day before two days after the day before tomorrow is Saturday. What day is it today?

2. What is the square root of 169?

3. Solve the equation 3y = 6y + 11 and find y.

4. There are two ducks in front of a duck, two ducks behind a duck, and a duck in the middle. How many ducks are there?

5. How many days does it take to travel from New York City to London by plane, assuming non-stop flights and average speeds?

6. What are the products of the chemical reaction between salicylic acid and acetic anhydride?

7. If five cats can catch five mice in five minutes, how long will it take one cat to catch one mouse?

8. Create a JS program that prints the first 100 Fibonacci numbers.
   '
   [1719977808] input tokens: [ '<|start_header_id|>':128006, 'user':882, '<|end_header_id|>':128007, '':271, 'Answer':16533, ' the':279, ' following':2768, ' questions':4860, ':':512, '1':16, '.':13, ' The':578, ' day':1938, ' before':1603, ' two':1403, ' days':2919, ' after':1306, ' the':279, ' day':1938, ' before':1603, ' tomorrow':16986, ' is':374, ' Saturday':7884, '.':13, ' What':3639, ' day':1938, ' is':374, ' it':433, ' today':3432, '?':5380, '2':17, '.':13, ' What':3639, ' is':374, ' the':279, ' square':9518, ' root':3789, ' of':315, ' ':220, '169':11739, '?':5380, '3':18, '.':13, ' Solve':64384, ' the':279, ' equation':24524, ' ':220, '3':18, 'y':88, ' =':284, ' ':220, '6':21, 'y':88, ' +':489, ' ':220, '11':806, ' and':323, ' find':1505, ' y':379, '.':627, '4':19, '.':13, ' There':2684, ' are':527, ' two':1403, ' ducks':78878, ' in':304, ' front':4156, ' of':315, ' a':264, ' duck':37085, ',':11, ' two':1403, ' ducks':78878, ' behind':4920, ' a':264, ' duck':37085, ',':11, ' and':323, ' a':264, ' duck':37085, ' in':304, ' the':279, ' middle':6278, '.':13, ' How':2650, ' many':1690, ' ducks':78878, ' are':527, ' there':1070, '?':5380, '5':20, '.':13, ' How':2650, ' many':1690, ' days':2919, ' does':1587, ' it':433, ' take':1935, ' to':311, ' travel':5944, ' from':505, ' New':1561, ' York':4356, ' City':4409, ' to':311, ' London':7295, ' by':555, ' plane':11277, ',':11, ' assuming':26619, ' non':2536, '-stop':42311, ' flights':25952, ' and':323, ' average':5578, ' speeds':25753, '?':5380, '6':21, '.':13, ' What':3639, ' are':527, ' the':279, ' products':3956, ' of':315, ' the':279, ' chemical':11742, ' reaction':13010, ' between':1990, ' sal':4371, 'icy':2912, 'lic':416, ' acid':13935, ' and':323, ' ac':1645, 'etic':5411, ' an':459, 'hy':8671, 'd':67, 'ride':1425, '?':5380, '7':22, '.':13, ' If':1442, ' five':4330, ' cats':19987, ' can':649, ' catch':2339, ' five':4330, ' mice':24548, ' in':304, ' five':4330, ' minutes':4520, ',':11, ' how':1268, ' long':1317, ' will':690, ' it':433, ' take':1935, ' one':832, ' cat':8415, ' to':311, ' catch':2339, ' one':832, ' mouse':8814, '?':5380, '8':23, '.':13, ' Create':4324, ' a':264, ' JS':12438, ' program':2068, ' that':430, ' prints':24370, ' the':279, ' first':1176, ' ':220, '100':1041, ' Fibonacci':80783, ' numbers':5219, '.':13, '<|eot_id|>':128009, '<|start_header_id|>':128006, 'assistant':78191, '<|end_header_id|>':128007, '':271 ]
   [1719977808] n_remain: -187
   [1719977808] eval: [ '<|start_header_id|>':128006, 'user':882 ]
   [1719977808] n_past = 8
   [1719977808] embd_inp.size(): 194, n_consumed: 8
   [1719977808] eval: [ '<|start_header_id|>':128006, 'user':882, '<|end_header_id|>':128007, '':271, 'Answer':16533, ' the':279, ' following':2768, ' questions':4860, ':':512, '1':16, '.':13, ' The':578, ' day':1938, ' before':1603, ' two':1403, ' days':2919, ' after':1306, ' the':279, ' day':1938, ' before':1603, ' tomorrow':16986, ' is':374, ' Saturday':7884, '.':13, ' What':3639, ' day':1938, ' is':374, ' it':433, ' today':3432, '?':5380, '2':17, '.':13, ' What':3639, ' is':374, ' the':279, ' square':9518, ' root':3789, ' of':315, ' ':220, '169':11739, '?':5380, '3':18, '.':13, ' Solve':64384, ' the':279, ' equation':24524, ' ':220, '3':18, 'y':88, ' =':284, ' ':220, '6':21, 'y':88, ' +':489, ' ':220, '11':806, ' and':323, ' find':1505, ' y':379, '.':627, '4':19, '.':13, ' There':2684, ' are':527, ' two':1403, ' ducks':78878, ' in':304, ' front':4156, ' of':315, ' a':264, ' duck':37085, ',':11, ' two':1403, ' ducks':78878, ' behind':4920, ' a':264, ' duck':37085, ',':11, ' and':323, ' a':264, ' duck':37085, ' in':304, ' the':279, ' middle':6278, '.':13, ' How':2650, ' many':1690, ' ducks':78878, ' are':527, ' there':1070, '?':5380, '5':20, '.':13, ' How':2650, ' many':1690, ' days':2919, ' does':1587, ' it':433, ' take':1935, ' to':311, ' travel':5944, ' from':505, ' New':1561, ' York':4356, ' City':4409, ' to':311, ' London':7295, ' by':555, ' plane':11277, ',':11, ' assuming':26619, ' non':2536, '-stop':42311, ' flights':25952, ' and':323, ' average':5578, ' speeds':25753, '?':5380, '6':21, '.':13, ' What':3639, ' are':527, ' the':279, ' products':3956, ' of':315, ' the':279, ' chemical':11742, ' reaction':13010, ' between':1990, ' sal':4371, 'icy':2912, 'lic':416, ' acid':13935, ' and':323, ' ac':1645, 'etic':5411, ' an':459, 'hy':8671, 'd':67, 'ride':1425, '?':5380, '7':22, '.':13, ' If':1442, ' five':4330, ' cats':19987, ' can':649, ' catch':2339, ' five':4330, ' mice':24548, ' in':304, ' five':4330, ' minutes':4520, ',':11, ' how':1268, ' long':1317, ' will':690, ' it':433, ' take':1935, ' one':832, ' cat':8415, ' to':311, ' catch':2339, ' one':832, ' mouse':8814, '?':5380, '8':23, '.':13, ' Create':4324, ' a':264, ' JS':12438, ' program':2068, ' that':430, ' prints':24370, ' the':279, ' first':1176, ' ':220, '100':1041, ' Fibonacci':80783, ' numbers':5219, '.':13, '<|eot_id|>':128009, '<|start_header_id|>':128006, 'assistant':78191, '<|end_header_id|>':128007, '':271 ]
   [1719977809] n_past = 194
   [1719977809] last: [ ' products':3956, ' of':315, ' the':279, ' chemical':11742, ' reaction':13010, ' between':1990, ' sal':4371, 'icy':2912, 'lic':416, ' acid':13935, ' and':323, ' ac':1645, 'etic':5411, ' an':459, 'hy':8671, 'd':67, 'ride':1425, '?':5380, '7':22, '.':13, ' If':1442, ' five':4330, ' cats':19987, ' can':649, ' catch':2339, ' five':4330, ' mice':24548, ' in':304, ' five':4330, ' minutes':4520, ',':11, ' how':1268, ' long':1317, ' will':690, ' it':433, ' take':1935, ' one':832, ' cat':8415, ' to':311, ' catch':2339, ' one':832, ' mouse':8814, '?':5380, '8':23, '.':13, ' Create':4324, ' a':264, ' JS':12438, ' program':2068, ' that':430, ' prints':24370, ' the':279, ' first':1176, ' ':220, '100':1041, ' Fibonacci':80783, ' numbers':5219, '.':13, '<|eot_id|>':128009, '<|start_header_id|>':128006, 'assistant':78191, '<|end_header_id|>':128007, '':271, 'I':40 ]
   [1719977809] n_remain: -188
   [1719977809] eval: [ 'I':40 ]
   [1719977809] n_past = 195
   [1719977809] last: [ ' of':315, ' the':279, ' chemical':11742, ' reaction':13010, ' between':1990, ' sal':4371, 'icy':2912, 'lic':416, ' acid':13935, ' and':323, ' ac':1645, 'etic':5411, ' an':459, 'hy':8671, 'd':67, 'ride':1425, '?':5380, '7':22, '.':13, ' If':1442, ' five':4330, ' cats':19987, ' can':649, ' catch':2339, ' five':4330, ' mice':24548, ' in':304, ' five':4330, ' minutes':4520, ',':11, ' how':1268, ' long':1317, ' will':690, ' it':433, ' take':1935, ' one':832, ' cat':8415, ' to':311, ' catch':2339, ' one':832, ' mouse':8814, '?':5380, '8':23, '.':13, ' Create':4324, ' a':264, ' JS':12438, ' program':2068, ' that':430, ' prints':24370, ' the':279, ' first':1176, ' ':220, '100':1041, ' Fibonacci':80783, ' numbers':5219, '.':13, '<|eot_id|>':128009, '<|start_header_id|>':128006, 'assistant':78191, '<|end_header_id|>':128007, '':271, 'I':40, ''d':4265 ]
   [1719977809] n_remain: -189
   [1719977809] eval: [ ''d':4265 ]
   [1719977809] n_past = 196
   [1719977809] last: [ ' the':279, ' chemical':11742, ' reaction':13010, ' between':1990, ' sal':4371, 'icy':2912, 'lic':416, ' acid':13935, ' and':323, ' ac':1645, 'etic':5411, ' an':459, 'hy':8671, 'd':67, 'ride':1425, '?':5380, '7':22, '.':13, ' If':1442, ' five':4330, ' cats':19987, ' can':649, ' catch':2339, ' five':4330, ' mice':24548, ' in':304, ' five':4330, ' minutes':4520, ',':11, ' how':1268, ' long':1317, ' will':690, ' it':433, ' take':1935, ' one':832, ' cat':8415, ' to':311, ' catch':2339, ' one':832, ' mouse':8814, '?':5380, '8':23, '.':13, ' Create':4324, ' a':264, ' JS':12438, ' program':2068, ' that':430, ' prints':24370, ' the':279, ' first':1176, ' ':220, '100':1041, ' Fibonacci':80783, ' numbers':5219, '.':13, '<|eot_id|>':128009, '<|start_header_id|>':128006, 'assistant':78191, '<|end_header_id|>':128007, '':271, 'I':40, ''d':4265, ' be':387 ]
   [1719977809] n_remain: -190
   [1719977809] eval: [ ' be':387 ]
   [1719977809] n_past = 197
   [1719977809] last: [ ' chemical':11742, ' reaction':13010, ' between':1990, ' sal':4371, 'icy':2912, 'lic':416, ' acid':13935, ' and':323, ' ac':1645, 'etic':5411, ' an':459, 'hy':8671, 'd':67, 'ride':1425, '?':5380, '7':22, '.':13, ' If':1442, ' five':4330, ' cats':19987, ' can':649, ' catch':2339, ' five':4330, ' mice':24548, ' in':304, ' five':4330, ' minutes':4520, ',':11, ' how':1268, ' long':1317, ' will':690, ' it':433, ' take':1935, ' one':832, ' cat':8415, ' to':311, ' catch':2339, ' one':832, ' mouse':8814, '?':5380, '8':23, '.':13, ' Create':4324, ' a':264, ' JS':12438, ' program':2068, ' that':430, ' prints':24370, ' the':279, ' first':1176, ' ':220, '100':1041, ' Fibonacci':80783, ' numbers':5219, '.':13, '<|eot_id|>':128009, '<|start_header_id|>':128006, 'assistant':78191, '<|end_header_id|>':128007, '':271, 'I':40, ''d':4265, ' be':387, ' happy':6380 ]
   [1719977809] n_remain: -191
   [1719977809] eval: [ ' happy':6380 ]
   [1719977809] n_past = 198
   [1719977809] last: [ ' reaction':13010, ' between':1990, ' sal':4371, 'icy':2912, 'lic':416, ' acid':13935, ' and':323, ' ac':1645, 'etic':5411, ' an':459, 'hy':8671, 'd':67, 'ride':1425, '?':5380, '7':22, '.':13, ' If':1442, ' five':4330, ' cats':19987, ' can':649, ' catch':2339, ' five':4330, ' mice':24548, ' in':304, ' five':4330, ' minutes':4520, ',':11, ' how':1268, ' long':1317, ' will':690, ' it':433, ' take':1935, ' one':832, ' cat':8415, ' to':311, ' catch':2339, ' one':832, ' mouse':8814, '?':5380, '8':23, '.':13, ' Create':4324, ' a':264, ' JS':12438, ' program':2068, ' that':430, ' prints':24370, ' the':279, ' first':1176, ' ':220, '100':1041, ' Fibonacci':80783, ' numbers':5219, '.':13, '<|eot_id|>':128009, '<|start_header_id|>':128006, 'assistant':78191, '<|end_header_id|>':128007, '':271, 'I':40, ''d':4265, ' be':387, ' happy':6380, ' to':311 ]
   [1719977809] n_remain: -192
   [1719977809] eval: [ ' to':311 ]
   [1719977809] n_past = 199
   [1719977809] last: [ ' between':1990, ' sal':4371, 'icy':2912, 'lic':416, ' acid':13935, ' and':323, ' ac':1645, 'etic':5411, ' an':459, 'hy':8671, 'd':67, 'ride':1425, '?':5380, '7':22, '.':13, ' If':1442, ' five':4330, ' cats':19987, ' can':649, ' catch':2339, ' five':4330, ' mice':24548, ' in':304, ' five':4330, ' minutes':4520, ',':11, ' how':1268, ' long':1317, ' will':690, ' it':433, ' take':1935, ' one':832, ' cat':8415, ' to':311, ' catch':2339, ' one':832, ' mouse':8814, '?':5380, '8':23, '.':13, ' Create':4324, ' a':264, ' JS':12438, ' program':2068, ' that':430, ' prints':24370, ' the':279, ' first':1176, ' ':220, '100':1041, ' Fibonacci':80783, ' numbers':5219, '.':13, '<|eot_id|>':128009, '<|start_header_id|>':128006, 'assistant':78191, '<|end_header_id|>':128007, '':271, 'I':40, ''d':4265, ' be':387, ' happy':6380, ' to':311, ' help':1520 ]
   [1719977809] n_remain: -193
   [1719977809] eval: [ ' help':1520 ]
   [1719977809] n_past = 200
   [1719977809]

                                    BLA BLA BLA 
   

�[31mTokens consumed so far = 4230 / 8192 �[0m
[1719978373] last: [ ' must':2011, ' be':387, ' Friday':6740, '!':2268, '2':17, '.':13, ' What':3639, ' is':374, ' the':279, ' square':9518, ' root':3789, ' of':315, ' ':220, '169':11739, '?':1980, 'The':791, ' square':9518, ' root':3789, ' of':315, ' ':220, '169':11739, ' is':374, ' ':220, '13':1032, ',':11, ' since':2533, ' ':220, '13':1032, ' ':25800, ' ':220, '13':1032, ' =':284, ' ':220, '169':11739, '.':382, '3':18, '.':13, ' Solve':64384, ' the':279, ' equation':24524, ' ':220, '3':18, 'y':88, ' =':284, ' ':220, '6':21, 'y':88, ' +':489, ' ':220, '11':806, ' and':323, ' find':1505, ' y':379, '.':382, 'Let':10267, ''s':596, ' rearr':56427, 'ange':853, ' the':279, ' equation':24524, ':':1473, '3':18, 'y':88, ' -':482 ]
[1719978373] n_remain: -4216
[1719978373] eval: [ ' -':482 ]
[1719978373] n_past = 4231
[1719978373] last: [ ' be':387, ' Friday':6740, '!':2268, '2':17, '.':13, ' What':3639, ' is':374, ' the':279, ' square':9518, ' root':3789, ' of':315, ' ':220, '169':11739, '?':1980, 'The':791, ' square':9518, ' root':3789, ' of':315, ' ':220, '169':11739, ' is':374, ' ':220, '13':1032, ',':11, ' since':2533, ' ':220, '13':1032, ' ':25800, ' ':220, '13':1032, ' =':284, ' ':220, '169':11739, '.':382, '3':18, '.':13, ' Solve':64384, ' the':279, ' equation':24524, ' ':220, '3':18, 'y':88, ' =':284, ' ':220, '6':21, 'y':88, ' +':489, ' ':220, '11':806, ' and':323, ' find':1505, ' y':379, '.':382, 'Let':10267, ''s':596, ' rearr':56427, 'ange':853, ' the':279, ' equation':24524, ':':1473, '3':18, 'y':88, ' -':482, ' ':220 ]
[1719978373] n_remain: -4217
[1719978373] eval: [ ' ':220 ]
[1719978373] n_past = 4232
[1719978373] last: [ ' Friday':6740, '!':2268, '2':17, '.':13, ' What':3639, ' is':374, ' the':279, ' square':9518, ' root':3789, ' of':315, ' ':220, '169':11739, '?':1980, 'The':791, ' square':9518, ' root':3789, ' of':315, ' ':220, '169':11739, ' is':374, ' ':220, '13':1032, ',':11, ' since':2533, ' ':220, '13':1032, ' ':25800, ' ':220, '13':1032, ' =':284, ' ':220, '169':11739, '.':382, '3':18, '.':13, ' Solve':64384, ' the':279, ' equation':24524, ' ':220, '3':18, 'y':88, ' =':284, ' ':220, '6':21, 'y':88, ' +':489, ' ':220, '11':806, ' and':323, ' find':1505, ' y':379, '.':382, 'Let':10267, ''s':596, ' rearr':56427, 'ange':853, ' the':279, ' equation':24524, ':':1473, '3':18, 'y':88, ' -':482, ' ':220, '6':21 ]
[1719978373] n_remain: -4218
[1719978373] eval: [ '6':21 ]
[1719978374] n_past = 4233
[1719978374] last: [ '!':2268, '2':17, '.':13, ' What':3639, ' is':374, ' the':279, ' square':9518, ' root':3789, ' of':315, ' ':220, '169':11739, '?':1980, 'The':791, ' square':9518, ' root':3789, ' of':315, ' ':220, '169':11739, ' is':374, ' ':220, '13':1032, ',':11, ' since':2533, ' ':220, '13':1032, ' ':25800, ' ':220, '13':1032, ' =':284, ' ':220, '169':11739, '.':382, '3':18, '.':13, ' Solve':64384, ' the':279, ' equation':24524, ' ':220, '3':18, 'y':88, ' =':284, ' ':220, '6':21, 'y':88, ' +':489, ' ':220, '11':806, ' and':323, ' find':1505, ' y':379, '.':382, 'Let':10267, ''s':596, ' rearr':56427, 'ange':853, ' the':279, ' equation':24524, ':':1473, '3':18, 'y':88, ' -':482, ' ':220, '6':21, 'y':88 ]
[1719978374] n_remain: -4219
[1719978374] eval: [ 'y':88 ]
[1719978374] n_past = 4234
[1719978374] last: [ '2':17, '.':13, ' What':3639, ' is':374, ' the':279, ' square':9518, ' root':3789, ' of':315, ' ':220, '169':11739, '?':1980, 'The':791, ' square':9518, ' root':3789, ' of':315, ' ':220, '169':11739, ' is':374, ' ':220, '13':1032, ',':11, ' since':2533, ' ':220, '13':1032, ' ':25800, ' ':220, '13':1032, ' =':284, ' ':220, '169':11739, '.':382, '3':18, '.':13, ' Solve':64384, ' the':279, ' equation':24524, ' ':220, '3':18, 'y':88, ' =':284, ' ':220, '6':21, 'y':88, ' +':489, ' ':220, '11':806, ' and':323, ' find':1505, ' y':379, '.':382, 'Let':10267, ''s':596, ' rearr':56427, 'ange':853, ' the':279, ' equation':24524, ':':1473, '3':18, 'y':88, ' -':482, ' ':220, '6':21, 'y':88, ' =':284 ]
[1719978374] n_remain: -4220
[1719978374] eval: [ ' =':284 ]
[1719978374] n_past = 4235
[1719978374] last: [ '.':13, ' What':3639, ' is':374, ' the':279, ' square':9518, ' root':3789, ' of':315, ' ':220, '169':11739, '?':1980, 'The':791, ' square':9518, ' root':3789, ' of':315, ' ':220, '169':11739, ' is':374, ' ':220, '13':1032, ',':11, ' since':2533, ' ':220, '13':1032, ' ':25800, ' ':220, '13':1032, ' =':284, ' ':220, '169':11739, '.':382, '3':18, '.':13, ' Solve':64384, ' the':279, ' equation':24524, ' ':220, '3':18, 'y':88, ' =':284, ' ':220, '6':21, 'y':88, ' +':489, ' ':220, '11':806, ' and':323, ' find':1505, ' y':379, '.':382, 'Let':10267, ''s':596, ' rearr':56427, 'ange':853, ' the':279, ' equation':24524, ':':1473, '3':18, 'y':88, ' -':482, ' ':220, '6':21, 'y':88, ' =':284, '<|eot_id|>':128009 ]
[1719978374] n_remain: -4221
[1719978374] found an EOG token
[1719978374] waiting for user input

                                         STOPS HERE for no reason then galucinates 
                      
                      
                      
         
                                                 using manualy formated prompt 

../llama.cpp/llama-cli --model ../../models/Meta-Llama-3-8B-Instruct_Q5_K_S.gguf --n-gpu-layers 25 -cnv --interactive --interactive-first --simple-io -b 2048 -n -1 -e --ctx_size 0 --temp 0.5 --top_k 10 --multiline-input --repeat_penalty 1.12 -t 8 -ptc 10 -r "<|start_header_id|>user" --logdir ./ --log-append --no-display-prompt

[1719975339] Log start
[1719975339] Cmd: ../llama.cpp/llama-cli --model ../../models/Meta-Llama-3-8B-Instruct_Q5_K_S.gguf --n-gpu-layers 25 -cnv --interactive --interactive-first --simple-io -b 2048 -n -1 -e --ctx_size 0 --temp 0.5 --top_k 10 --multiline-input --repeat_penalty 1.12 -t 8 -ptc 10 -r <|start_header_id|>user --logdir ./ --log-append --no-display-prompt
[1719975339] main: build = 0 (unknown)
[1719975339] main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
[1719975339] main: seed = 1719975339
[1719975339] main: llama backend init
[1719975339] main: load the model and apply lora adapter, if any
[1719975339] llama_model_loader: loaded meta data with 22 key-value pairs and 291 tensors from ../../models/Meta-Llama-3-8B-Instruct_Q5_K_S.gguf (version GGUF V3 (latest))
[1719975339] llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
[1719975339] llama_model_loader: - kv 0: general.architecture str = llama
[1719975339] llama_model_loader: - kv 1: general.name str = models
[1719975339] llama_model_loader: - kv 2: llama.block_count u32 = 32
[1719975339] llama_model_loader: - kv 3: llama.context_length u32 = 8192
[1719975339] llama_model_loader: - kv 4: llama.embedding_length u32 = 4096
[1719975339] llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336
[1719975339] llama_model_loader: - kv 6: llama.attention.head_count u32 = 32
[1719975339] llama_model_loader: - kv 7: llama.attention.head_count_kv u32 = 8
[1719975339] llama_model_loader: - kv 8: llama.rope.freq_base f32 = 500000.000000
[1719975339] llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
[1719975339] llama_model_loader: - kv 10: general.file_type u32 = 16
[1719975339] llama_model_loader: - kv 11: llama.vocab_size u32 = 128256
[1719975339] llama_model_loader: - kv 12: llama.rope.dimension_count u32 = 128
[1719975339] llama_model_loader: - kv 13: tokenizer.ggml.model str = gpt2
[1719975339] llama_model_loader: - kv 14: tokenizer.ggml.pre str = llama-bpe
[1719975339] llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,128256] = ["!", """, "#", "$", "%", "&", "'", ...
[1719975339] llama_model_loader: - kv 16: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
[1719975340] llama_model_loader: - kv 17: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
[1719975340] llama_model_loader: - kv 18: tokenizer.ggml.bos_token_id u32 = 128000
[1719975340] llama_model_loader: - kv 19: tokenizer.ggml.eos_token_id u32 = 128001
[1719975340] llama_model_loader: - kv 20: tokenizer.chat_template str = {% set loop_messages = messages %}{% ...
[1719975340] llama_model_loader: - kv 21: general.quantization_version u32 = 2
[1719975340] llama_model_loader: - type f32: 65 tensors
[1719975340] llama_model_loader: - type q5_K: 225 tensors
[1719975340] llama_model_loader: - type q6_K: 1 tensors
[1719975340] llm_load_vocab: special tokens cache size = 256
[1719975340] llm_load_vocab: token to piece cache size = 0.8000 MB
[1719975340] llm_load_print_meta: format = GGUF V3 (latest)
[1719975340] llm_load_print_meta: arch = llama
[1719975340] llm_load_print_meta: vocab type = BPE
[1719975340] llm_load_print_meta: n_vocab = 128256
[1719975340] llm_load_print_meta: n_merges = 280147
[1719975340] llm_load_print_meta: n_ctx_train = 8192
[1719975340] llm_load_print_meta: n_embd = 4096
[1719975340] llm_load_print_meta: n_head = 32
[1719975340] llm_load_print_meta: n_head_kv = 8
[1719975340] llm_load_print_meta: n_layer = 32
[1719975340] llm_load_print_meta: n_rot = 128
[1719975340] llm_load_print_meta: n_swa = 0
[1719975340] llm_load_print_meta: n_embd_head_k = 128
[1719975340] llm_load_print_meta: n_embd_head_v = 128
[1719975340] llm_load_print_meta: n_gqa = 4
[1719975340] llm_load_print_meta: n_embd_k_gqa = 1024
[1719975340] llm_load_print_meta: n_embd_v_gqa = 1024
[1719975340] llm_load_print_meta: f_norm_eps = 0.0e+00
[1719975340] llm_load_print_meta: f_norm_rms_eps = 1.0e-05
[1719975340] llm_load_print_meta: f_clamp_kqv = 0.0e+00
[1719975340] llm_load_print_meta: f_max_alibi_bias = 0.0e+00
[1719975340] llm_load_print_meta: f_logit_scale = 0.0e+00
[1719975340] llm_load_print_meta: n_ff = 14336
[1719975340] llm_load_print_meta: n_expert = 0
[1719975340] llm_load_print_meta: n_expert_used = 0
[1719975340] llm_load_print_meta: causal attn = 1
[1719975340] llm_load_print_meta: pooling type = 0
[1719975340] llm_load_print_meta: rope type = 0
[1719975340] llm_load_print_meta: rope scaling = linear
[1719975340] llm_load_print_meta: freq_base_train = 500000.0
[1719975340] llm_load_print_meta: freq_scale_train = 1
[1719975340] llm_load_print_meta: n_ctx_orig_yarn = 8192
[1719975340] llm_load_print_meta: rope_finetuned = unknown
[1719975340] llm_load_print_meta: ssm_d_conv = 0
[1719975340] llm_load_print_meta: ssm_d_inner = 0
[1719975340] llm_load_print_meta: ssm_d_state = 0
[1719975340] llm_load_print_meta: ssm_dt_rank = 0
[1719975340] llm_load_print_meta: model type = 8B
[1719975340] llm_load_print_meta: model ftype = Q5_K - Small
[1719975340] llm_load_print_meta: model params = 8.03 B
[1719975340] llm_load_print_meta: model size = 5.21 GiB (5.57 BPW)
[1719975340] llm_load_print_meta: general.name = models
[1719975340] llm_load_print_meta: BOS token = 128000 '<|begin_of_text|>'
[1719975340] llm_load_print_meta: EOS token = 128001 '<|end_of_text|>'
[1719975340] llm_load_print_meta: LF token = 128 'Ä'
[1719975340] llm_load_print_meta: EOT token = 128009 '<|eot_id|>'
[1719975340] llm_load_print_meta: max token length = 256
[1719975340] ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
[1719975340] ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
[1719975340] ggml_cuda_init: found 1 CUDA devices:
[1719975340] Device 0: NVIDIA GeForce GTX 1060 6GB, compute capability 6.1, VMM: yes
[1719975340] llm_load_tensors: ggml ctx size = 0.27 MiB
[1719975341] llm_load_tensors: offloading 25 repeating layers to GPU
[1719975341] llm_load_tensors: offloaded 25/33 layers to GPU
[1719975341] llm_load_tensors: CPU buffer size = 5332.43 MiB
[1719975341] llm_load_tensors: CUDA0 buffer size = 3575.78 MiB
[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341] .[1719975341]
[1719975341] llama_new_context_with_model: n_ctx = 8192
[1719975341] llama_new_context_with_model: n_batch = 2048
[1719975341] llama_new_context_with_model: n_ubatch = 512
[1719975341] llama_new_context_with_model: flash_attn = 0
[1719975341] llama_new_context_with_model: freq_base = 500000.0
[1719975341] llama_new_context_with_model: freq_scale = 1
[1719975341] llama_kv_cache_init: CUDA_Host KV buffer size = 224.00 MiB
[1719975341] llama_kv_cache_init: CUDA0 KV buffer size = 800.00 MiB
[1719975341] llama_new_context_with_model: KV self size = 1024.00 MiB, K (f16): 512.00 MiB, V (f16): 512.00 MiB
[1719975341] llama_new_context_with_model: CUDA_Host output buffer size = 0.49 MiB
[1719975341] llama_new_context_with_model: CUDA0 compute buffer size = 669.48 MiB
[1719975341] llama_new_context_with_model: CUDA_Host compute buffer size = 24.01 MiB
[1719975341] llama_new_context_with_model: graph nodes = 1030
[1719975341] llama_new_context_with_model: graph splits = 81
[1719975341] warming up the model with an empty run
[1719975342] n_ctx: 8192
[1719975342] main: chat template example: <|start_header_id|>system<|end_header_id|>

You are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>

Hello<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Hi there<|eot_id|><|start_header_id|>user<|end_header_id|>

How are you?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

[1719975342]
[1719975342] system_info: n_threads = 8 / 16 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
[1719975342] add_bos: 1
[1719975342] tokenize the prompt
[1719975342] prompt: "<|start_header_id|>system<|end_header_id|>

<|eot_id|>"
[1719975342] tokens: [ '<|begin_of_text|>':128000, '<|start_header_id|>':128006, 'system':9125, '<|end_header_id|>':128007, '':271, '<|eot_id|>':128009 ]
[1719975342] recalculate the cached logits (check): embd_inp.empty() false, n_matching_session_tokens 0, embd_inp.size() 6, session_tokens.size() 0, embd_inp.size() 6
[1719975342] main: interactive mode on.
[1719975342] Reverse prompt: '<|start_header_id|>user'
[1719975342] sampling:
repeat_last_n = 64, repeat_penalty = 1.120, frequency_penalty = 0.000, presence_penalty = 0.000
top_k = 10, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.500
mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
[1719975342] sampling order:
CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temperature
[1719975342] generate: n_ctx = 8192, n_batch = 2048, n_predict = -1, n_keep = 1
[1719975342]

[1719975342] == Running in interactive mode. ==
[1719975342] - Press Ctrl+C to interject at any time.
[1719975342] - To return control to the AI, end your input with ''.

• To return control without starting a new line, end your input with '/'.

[1719975342] embd_inp.size(): 6, n_consumed: 0
[1719975342] found an EOG token
[1719975342] eval: [ '<|begin_of_text|>':128000, '<|start_header_id|>':128006, 'system':9125, '<|end_header_id|>':128007, '':271, '<|eot_id|>':128009 ]
[1719975342] n_past = 6
[1719975342] embd_inp.size(): 8, n_consumed: 6
[1719975342] found antiprompt: !!!!!!!!!!!!!!!!!!!!!!!!<|begin_of_text|><|start_header_id|>system<|end_header_id|>

<|eot_id|><|start_header_id|>user
[1719975342] waiting for user input
[1719975346] buffer: '<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Role and Purpose: You are Alice, a large language model. Your purpose is to assist users by providing information, answering questions, and engaging in meaningful conversations based on the data you were trained on. Behavior and Tone: Be informative, engaging, and respectful. Maintain a neutral and unbiased tone. Ensure that responses are clear and concise. Capabilities: Use your training data to provide accurate and relevant information. Explain complex concepts in an easy-to-understand manner.Provide sources when referencing specific information or data. Output Formatting: Use this formatting for code: language /n<|eot_id|><|start_header_id|>user<|end_header_id|>

Answer the following questions:

1. The day before two days after the day before tomorrow is Saturday. What day is it today?
2. What is the square root of 169?
3. Solve the equation 3y = 6y + 11 and find y.
4. There are two ducks in front of a duck, two ducks behind a duck, and a duck in the middle. How many ducks are there?
5. How many days does it take to travel from New York City to London by plane, assuming non-stop flights and average speeds?
6. What are the products of the chemical reaction between salicylic acid and acetic anhydride?
7. If five cats can catch five mice in five minutes, how long will it take one cat to catch one mouse?
8. Create a JS program that prints the first 100 Fibonacci numbers. <|eot_id|><|start_header_id|>assistant<|end_header_id|>

'
[1719975346] input tokens: [ '<|start_header_id|>':128006, 'user':882, '<|end_header_id|>':128007, '':271, '<|begin_of_text|>':128000, '<|start_header_id|>':128006, 'system':9125, '<|end_header_id|>':128007, '':271, 'Role':9207, ' and':323, ' Purpose':30145, ':':25, ' You':1472, ' are':527, ' Alice':30505, ',':11, ' a':264, ' large':3544, ' language':4221, ' model':1646, '.':13, ' Your':4718, ' purpose':7580, ' is':374, ' to':311, ' assist':7945, ' users':3932, ' by':555, ' providing':8405, ' information':2038, ',':11, ' answering':36864, ' questions':4860, ',':11, ' and':323, ' engaging':23387, ' in':304, ' meaningful':23222, ' conversations':21633, ' based':3196, ' on':389, ' the':279, ' data':828, ' you':499, ' were':1051, ' trained':16572, ' on':389, '.':13, ' Behavior':27773, ' and':323, ' Tone':68004, ':':25, ' ':220, ' Be':2893, ' informative':39319, ',':11, ' engaging':23387, ',':11, ' and':323, ' respectful':49150, '.':13, ' Maintain':87477, ' a':264, ' neutral':21277, ' and':323, ' unbiased':74315, ' tone':16630, '.':13, ' Ensure':30379, ' that':430, ' responses':14847, ' are':527, ' clear':2867, ' and':323, ' concise':64694, '.':13, ' Cap':8171, 'abilities':8623, ':':25, ' Use':5560, ' your':701, ' training':4967, ' data':828, ' to':311, ' provide':3493, ' accurate':13687, ' and':323, ' relevant':9959, ' information':2038, '.':13, ' Explain':83017, ' complex':6485, ' concepts':19476, ' in':304, ' an':459, ' easy':4228, '-to':4791, '-under':72207, 'stand':2752, ' manner':11827, '.Pro':7919, 'vide':20013, ' sources':8336, ' when':994, ' referencing':57616, ' specific':3230, ' information':2038, ' or':477, ' data':828, '.':13, ' Output':9442, ' Formatting':90688, ':':25, ' Use':5560, ' this':420, ' formatting':37666, ' for':369, ' code':2082, ':':25, ' ':55375, 'language':11789, ' /':611, 'n':77, '':74694, '<|eot_id|>':128009, '<|start_header_id|>':128006, 'user':882, '<|end_header_id|>':128007, '':271, 'Answer':16533, ' the':279, ' following':2768, ' questions':4860, ':':512, '1':16, '.':13, ' The':578, ' day':1938, ' before':1603, ' two':1403, ' days':2919, ' after':1306, ' the':279, ' day':1938, ' before':1603, ' tomorrow':16986, ' is':374, ' Saturday':7884, '.':13, ' What':3639, ' day':1938, ' is':374, ' it':433, ' today':3432, '?':5380, '2':17, '.':13, ' What':3639, ' is':374, ' the':279, ' square':9518, ' root':3789, ' of':315, ' ':220, '169':11739, '?':5380, '3':18, '.':13, ' Solve':64384, ' the':279, ' equation':24524, ' ':220, '3':18, 'y':88, ' =':284, ' ':220, '6':21, 'y':88, ' +':489, ' ':220, '11':806, ' and':323, ' find':1505, ' y':379, '.':627, '4':19, '.':13, ' There':2684, ' are':527, ' two':1403, ' ducks':78878, ' in':304, ' front':4156, ' of':315, ' a':264, ' duck':37085, ',':11, ' two':1403, ' ducks':78878, ' behind':4920, ' a':264, ' duck':37085, ',':11, ' and':323, ' a':264, ' duck':37085, ' in':304, ' the':279, ' middle':6278, '.':13, ' How':2650, ' many':1690, ' ducks':78878, ' are':527, ' there':1070, '?':5380, '5':20, '.':13, ' How':2650, ' many':1690, ' days':2919, ' does':1587, ' it':433, ' take':1935, ' to':311, ' travel':5944, ' from':505, ' New':1561, ' York':4356, ' City':4409, ' to':311, ' London':7295, ' by':555, ' plane':11277, ',':11, ' assuming':26619, ' non':2536, '-stop':42311, ' flights':25952, ' and':323, ' average':5578, ' speeds':25753, '?':5380, '6':21, '.':13, ' What':3639, ' are':527, ' the':279, ' products':3956, ' of':315, ' the':279, ' chemical':11742, ' reaction':13010, ' between':1990, ' sal':4371, 'icy':2912, 'lic':416, ' acid':13935, ' and':323, ' ac':1645, 'etic':5411, ' an':459, 'hy':8671, 'd':67, 'ride':1425, '?':5380, '7':22, '.':13, ' If':1442, ' five':4330, ' cats':19987, ' can':649, ' catch':2339, ' five':4330, ' mice':24548, ' in':304, ' five':4330, ' minutes':4520, ',':11, ' how':1268, ' long':1317, ' will':690, ' it':433, ' take':1935, ' one':832, ' cat':8415, ' to':311, ' catch':2339, ' one':832, ' mouse':8814, '?':5380, '8':23, '.':13, ' Create':4324, ' a':264, ' JS':12438, ' program':2068, ' that':430, ' prints':24370, ' the':279, ' first':1176, ' ':220, '100':1041, ' Fibonacci':80783, ' numbers':5219, '.':13, ' ':256, '<|eot_id|>':128009, '<|start_header_id|>':128006, 'assistant':78191, '<|end_header_id|>':128007, '<|eot_id|>':128009, '<|start_header_id|>':128006, 'assistant':78191, '<|end_header_id|>':128007, '':271 ]
[1719975346] n_remain: -318
[1719975346] eval: [ '<|start_header_id|>':128006, 'user':882 ]
[1719975347] n_past = 8
[1719975347] embd_inp.size(): 325, n_consumed: 8
[1719975347] eval: [ '<|start_header_id|>':128006, 'user':882, '<|end_header_id|>':128007, '':271, '<|begin_of_text|>':128000, '<|start_header_id|>':128006, 'system':9125, '<|end_header_id|>':128007, '':271, 'Role':9207, ' and':323, ' Purpose':30145, ':':25, ' You':1472, ' are':527, ' Alice':30505, ',':11, ' a':264, ' large':3544, ' language':4221, ' model':1646, '.':13, ' Your':4718, ' purpose':7580, ' is':374, ' to':311, ' assist':7945, ' users':3932, ' by':555, ' providing':8405, ' information':2038, ',':11, ' answering':36864, ' questions':4860, ',':11, ' and':323, ' engaging':23387, ' in':304, ' meaningful':23222, ' conversations':21633, ' based':3196, ' on':389, ' the':279, ' data':828, ' you':499, ' were':1051, ' trained':16572, ' on':389, '.':13, ' Behavior':27773, ' and':323, ' Tone':68004, ':':25, ' ':220, ' Be':2893, ' informative':39319, ',':11, ' engaging':23387, ',':11, ' and':323, ' respectful':49150, '.':13, ' Maintain':87477, ' a':264, ' neutral':21277, ' and':323, ' unbiased':74315, ' tone':16630, '.':13, ' Ensure':30379, ' that':430, ' responses':14847, ' are':527, ' clear':2867, ' and':323, ' concise':64694, '.':13, ' Cap':8171, 'abilities':8623, ':':25, ' Use':5560, ' your':701, ' training':4967, ' data':828, ' to':311, ' provide':3493, ' accurate':13687, ' and':323, ' relevant':9959, ' information':2038, '.':13, ' Explain':83017, ' complex':6485, ' concepts':19476, ' in':304, ' an':459, ' easy':4228, '-to':4791, '-under':72207, 'stand':2752, ' manner':11827, '.Pro':7919, 'vide':20013, ' sources':8336, ' when':994, ' referencing':57616, ' specific':3230, ' information':2038, ' or':477, ' data':828, '.':13, ' Output':9442, ' Formatting':90688, ':':25, ' Use':5560, ' this':420, ' formatting':37666, ' for':369, ' code':2082, ':':25, ' ':55375, 'language':11789, ' /':611, 'n':77, '':74694, '<|eot_id|>':128009, '<|start_header_id|>':128006, 'user':882, '<|end_header_id|>':128007, '':271, 'Answer':16533, ' the':279, ' following':2768, ' questions':4860, ':':512, '1':16, '.':13, ' The':578, ' day':1938, ' before':1603, ' two':1403, ' days':2919, ' after':1306, ' the':279, ' day':1938, ' before':1603, ' tomorrow':16986, ' is':374, ' Saturday':7884, '.':13, ' What':3639, ' day':1938, ' is':374, ' it':433, ' today':3432, '?':5380, '2':17, '.':13, ' What':3639, ' is':374, ' the':279, ' square':9518, ' root':3789, ' of':315, ' ':220, '169':11739, '?':5380, '3':18, '.':13, ' Solve':64384, ' the':279, ' equation':24524, ' ':220, '3':18, 'y':88, ' =':284, ' ':220, '6':21, 'y':88, ' +':489, ' ':220, '11':806, ' and':323, ' find':1505, ' y':379, '.':627, '4':19, '.':13, ' There':2684, ' are':527, ' two':1403, ' ducks':78878, ' in':304, ' front':4156, ' of':315, ' a':264, ' duck':37085, ',':11, ' two':1403, ' ducks':78878, ' behind':4920, ' a':264, ' duck':37085, ',':11, ' and':323, ' a':264, ' duck':37085, ' in':304, ' the':279, ' middle':6278, '.':13, ' How':2650, ' many':1690, ' ducks':78878, ' are':527, ' there':1070, '?':5380, '5':20, '.':13, ' How':2650, ' many':1690, ' days':2919, ' does':1587, ' it':433, ' take':1935, ' to':311, ' travel':5944, ' from':505, ' New':1561, ' York':4356, ' City':4409, ' to':311, ' London':7295, ' by':555, ' plane':11277, ',':11, ' assuming':26619, ' non':2536, '-stop':42311, ' flights':25952, ' and':323, ' average':5578, ' speeds':25753, '?':5380, '6':21, '.':13, ' What':3639, ' are':527, ' the':279, ' products':3956, ' of':315, ' the':279, ' chemical':11742, ' reaction':13010, ' between':1990, ' sal':4371, 'icy':2912, 'lic':416, ' acid':13935, ' and':323, ' ac':1645, 'etic':5411, ' an':459, 'hy':8671, 'd':67, 'ride':1425, '?':5380, '7':22, '.':13, ' If':1442, ' five':4330, ' cats':19987, ' can':649, ' catch':2339, ' five':4330, ' mice':24548, ' in':304, ' five':4330, ' minutes':4520, ',':11, ' how':1268, ' long':1317, ' will':690, ' it':433, ' take':1935, ' one':832, ' cat':8415, ' to':311, ' catch':2339, ' one':832, ' mouse':8814, '?':5380, '8':23, '.':13, ' Create':4324, ' a':264, ' JS':12438, ' program':2068, ' that':430, ' prints':24370, ' the':279, ' first':1176, ' ':220, '100':1041, ' Fibonacci':80783, ' numbers':5219, '.':13, ' ':256, '<|eot_id|>':128009, '<|start_header_id|>':128006, 'assistant':78191, '<|end_header_id|>':128007, '<|eot_id|>':128009, '<|start_header_id|>':128006, 'assistant':78191, '<|end_header_id|>':128007, '':271 ]
[1719975348] n_past = 325
[1719975348] last: [ ' between':1990, ' sal':4371, 'icy':2912, 'lic':416, ' acid':13935, ' and':323, ' ac':1645, 'etic':5411, ' an':459, 'hy':8671, 'd':67, 'ride':1425, '?':5380, '7':22, '.':13, ' If':1442, ' five':4330, ' cats':19987, ' can':649, ' catch':2339, ' five':4330, ' mice':24548, ' in':304, ' five':4330, ' minutes':4520, ',':11, ' how':1268, ' long':1317, ' will':690, ' it':433, ' take':1935, ' one':832, ' cat':8415, ' to':311, ' catch':2339, ' one':832, ' mouse':8814, '?':5380, '8':23, '.':13, ' Create':4324, ' a':264, ' JS':12438, ' program':2068, ' that':430, ' prints':24370, ' the':279, ' first':1176, ' ':220, '100':1041, ' Fibonacci':80783, ' numbers':5219, '.':13, ' ':256, '<|eot_id|>':128009, '<|start_header_id|>':128006, 'assistant':78191, '<|end_header_id|>':128007, '<|eot_id|>':128009, '<|start_header_id|>':128006, 'assistant':78191, '<|end_header_id|>':128007, '':271, 'I':40 ]
[1719975348] n_remain: -319
[1719975348] eval: [ 'I':40 ]
[1719975348] n_past = 326
[1719975348] last: [ ' sal':4371, 'icy':2912, 'lic':416, ' acid':13935, ' and':323, ' ac':1645, 'etic':5411, ' an':459, 'hy':8671, 'd':67, 'ride':1425, '?':5380, '7':22, '.':13, ' If':1442, ' five':4330, ' cats':19987, ' can':649, ' catch':2339, ' five':4330, ' mice':24548, ' in':304, ' five':4330, ' minutes':4520, ',':11, ' how':1268, ' long':1317, ' will':690, ' it':433, ' take':1935, ' one':832, ' cat':8415, ' to':311, ' catch':2339, ' one':832, ' mouse':8814, '?':5380, '8':23, '.':13, ' Create':4324, ' a':264, ' JS':12438, ' program':2068, ' that':430, ' prints':24370, ' the':279, ' first':1176, ' ':220, '100':1041, ' Fibonacci':80783, ' numbers':5219, '.':13, ' ':256, '<|eot_id|>':128009, '<|start_header_id|>':128006, 'assistant':78191, '<|end_header_id|>':128007, '<|eot_id|>':128009, '<|start_header_id|>':128006, 'assistant':78191, '<|end_header_id|>':128007, '':271, 'I':40, ''d':4265 ]
[1719975348] n_remain: -320
[1719975348] eval: [ ''d':4265 ]
[1719975348] n_past = 327
[1719975348] last: [ 'icy':2912, 'lic':416, ' acid':13935, ' and':323, ' ac':1645, 'etic':5411, ' an':459, 'hy':8671, 'd':67, 'ride':1425, '?':5380, '7':22, '.':13, ' If':1442, ' five':4330, ' cats':19987, ' can':649, ' catch':2339, ' five':4330, ' mice':24548, ' in':304, ' five':4330, ' minutes':4520, ',':11, ' how':1268, ' long':1317, ' will':690, ' it':433, ' take':1935, ' one':832, ' cat':8415, ' to':311, ' catch':2339, ' one':832, ' mouse':8814, '?':5380, '8':23, '.':13, ' Create':4324, ' a':264, ' JS':12438, ' program':2068, ' that':430, ' prints':24370, ' the':279, ' first':1176, ' ':220, '100':1041, ' Fibonacci':80783, ' numbers':5219, '.':13, ' ':256, '<|eot_id|>':128009, '<|start_header_id|>':128006, 'assistant':78191, '<|end_header_id|>':128007, '<|eot_id|>':128009, '<|start_header_id|>':128006, 'assistant':78191, '<|end_header_id|>':128007, '':271, 'I':40, ''d':4265, ' be':387 ]
[1719975348] n_remain: -321
[1719975348] eval: [ ' be':387 ]
[1719975348] n_past = 328
[1719975348] last: [ 'lic':416, ' acid':13935, ' and':323, ' ac':1645, 'etic':5411, ' an':459, 'hy':8671, 'd':67, 'ride':1425, '?':5380, '7':22, '.':13, ' If':1442, ' five':4330, ' cats':19987, ' can':649, ' catch':2339, ' five':4330, ' mice':24548, ' in':304, ' five':4330, ' minutes':4520, ',':11, ' how':1268, ' long':1317, ' will':690, ' it':433, ' take':1935, ' one':832, ' cat':8415, ' to':311, ' catch':2339, ' one':832, ' mouse':8814, '?':5380, '8':23, '.':13, ' Create':4324, ' a':264, ' JS':12438, ' program':2068, ' that':430, ' prints':24370, ' the':279, ' first':1176, ' ':220, '100':1041, ' Fibonacci':80783, ' numbers':5219, '.':13, ' ':256, '<|eot_id|>':128009, '<|start_header_id|>':128006, 'assistant':78191, '<|end_header_id|>':128007, '<|eot_id|>':128009, '<|start_header_id|>':128006, 'assistant':78191, '<|end_header_id|>':128007, '':271, 'I':40, ''d':4265, ' be':387, ' happy':6380 ]
[1719975348] n_remain: -322
[1719975348] eval: [ ' happy':6380 ]
[1719975348] n_past = 329
[1719975348] last: [ ' acid':13935, ' and':323, ' ac':1645, 'etic':5411, ' an':459, 'hy':8671, 'd':67, 'ride':1425, '?':5380, '7':22, '.':13, ' If':1442, ' five':4330, ' cats':19987, ' can':649, ' catch':2339, ' five':4330, ' mice':24548, ' in':304, ' five':4330, ' minutes':4520, ',':11, ' how':1268, ' long':1317, ' will':690, ' it':433, ' take':1935, ' one':832, ' cat':8415, ' to':311, ' catch':2339, ' one':832, ' mouse':8814, '?':5380, '8':23, '.':13, ' Create':4324, ' a':264, ' JS':12438, ' program':2068, ' that':430, ' prints':24370, ' the':279, ' first':1176, ' ':220, '100':1041, ' Fibonacci':80783, ' numbers':5219, '.':13, ' ':256, '<|eot_id|>':128009, '<|start_header_id|>':128006, 'assistant':78191, '<|end_header_id|>':128007, '<|eot_id|>':128009, '<|start_header_id|>':128006, 'assistant':78191, '<|end_header_id|>':128007, '':271, 'I':40, ''d':4265, ' be':387, ' happy':6380, ' to':311 ]
[1719975348] n_remain: -323
[1719975348] eval: [ ' to':311 ]
[1719975348] n_past = 330
[1719975348]
�[31mTokens consumed so far = 330 / 8192 �[0m

            ----------- This goes on forever ----------------
                      ` 

[ngxson]

The reason why you got this result is because only one system: chat template OR inprefix/suffix system can exist at the same time.

Instead of adding a lot of options, you should try with simple command at once to isolate the problem:

../llama.cpp/llama-cli --model ../../models/Meta-Llama-3-8B-Instruct_Q5_K_S.gguf -cnv
../llama.cpp/llama-cli --model ../../models/Meta-Llama-3-8B-Instruct_Q5_K_S.gguf -cnv --chat-template llama3
../llama.cpp/llama-cli --model ../../models/Meta-Llama-3-8B-Instruct_Q5_K_S.gguf -cnv --chat-template llama3 --no-display-prompt

...

(Hide your comment because it's too long, consider using https://gist.github.com/scmx/eca72d44afee0113ceb0349dd54a84a2 if you post long comments)

[dspasyuk]

@ngxson Than kyou for your help. Running just this ../llama.cpp/llama-cli --model ../../models/Meta-Llama-3-8B-Instruct_Q5_K_S.gguf --n-gpu-layers 25 -cnv --multiline-input --chat-template llama3

Producing about the same result as this: ../llama.cpp/llama-cli --model ../../models/Meta-Llama-3-8B-Instruct_Q5_K_S.gguf --n-gpu-layers 25 -cnv --interactive --interactive-first --simple-io -b 2048 -n -1 -e --ctx_size 0 --temp 0.5 --top_k 10 --multiline-input --repeat_penalty 1.12 -t 8 -ptc 10 --chat-template llama3 -r "<|start_header_id|>user" --logdir ./ --log-append --no-display-prompt

Generation eventually just stops after about 5000 tokens but resumes no issue and does not hallucinate. The difference I see between manual prompting and llama3 template is that in the template version, it uses bos token only at the beginning and in the manual prompting it looks something like this: input tokens: [ '<|start_header_id|>':128006, 'user':882, '<|end_header_id|>':128007, '':271, '<|begin_of_text|>':128000, '<|start_header_id|>':128006, 'system':9125, '<|end_header_id|>':128007, '':271, 'Role':9207, ' and':323, ' Purpose':30145, ':':25, ' You':1472, ' are':527, ' Alice':30505, ',':11, ' a':264, ' large':3544, ' language':4221, ' model':1646,

And I do not instert these: [ '<|start_header_id|>':128006, 'user':882, '<|end_header_id|>':128007, '':271,

I starts the message here: '<|begin_of_text|>':128000, '<|start_header_id|>':128006, 'system':9125, '<|end_header_id|>':128007, '':271, 'Role':9207, ' and':323, ' Purpose':30145, ':':25, ' You':1472, ' are':527, ' Alice':30505, ',':11, ' a':264, ' large':3544, ' language':4221, ' model':1646,