You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The llama.cpp tokenizer currently adds an EOS token unconditionally. However, adding an EOS token to the end of the system prompt is necessary to prevent generation before user input. This leads to two EOS tokens before the user prompt, potentially causing suboptimal performance.
llama.cpp-b2972, MacOS 14.5
./main -m Meta-Llama-3-8B-Instruct-Q8_0.gguf --temp 0 -i -e -p "<|start_header_id|>system<|end_header_id|>\n\nYou are a helpful assistant.<|eot_id|>" -r "<|eot_id|>" --in-prefix "<|start_header_id|>user<|end_header_id|>\n\n" --in-suffix "<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
Current behavior:
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful assistant.<|eot_id|>
<|eot_id|>
<|start_header_id|>user<|end_header_id|>
Expected behavior:
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful assistant.<|eot_id|>
<|start_header_id|>user<|end_header_id|>
The text was updated successfully, but these errors were encountered:
Also -ins is not used for llama3, it's designed to work with Alpaca model
-ins, --instruct: Run the program in instruction mode, which is specifically designed to work with Alpaca models that excel in completing tasks based on user instructions.
Uh oh!
There was an error while loading. Please reload this page.
The llama.cpp tokenizer currently adds an EOS token unconditionally. However, adding an EOS token to the end of the system prompt is necessary to prevent generation before user input. This leads to two EOS tokens before the user prompt, potentially causing suboptimal performance.
llama.cpp-b2972, MacOS 14.5
./main -m Meta-Llama-3-8B-Instruct-Q8_0.gguf --temp 0 -i -e -p "<|start_header_id|>system<|end_header_id|>\n\nYou are a helpful assistant.<|eot_id|>" -r "<|eot_id|>" --in-prefix "<|start_header_id|>user<|end_header_id|>\n\n" --in-suffix "<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
Current behavior:
Expected behavior:
The text was updated successfully, but these errors were encountered: