Skip to content

tokenization: double EOS tokens #7484

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Azirine opened this issue May 23, 2024 · 2 comments
Closed

tokenization: double EOS tokens #7484

Azirine opened this issue May 23, 2024 · 2 comments

Comments

@Azirine
Copy link

Azirine commented May 23, 2024

The llama.cpp tokenizer currently adds an EOS token unconditionally. However, adding an EOS token to the end of the system prompt is necessary to prevent generation before user input. This leads to two EOS tokens before the user prompt, potentially causing suboptimal performance.

llama.cpp-b2972, MacOS 14.5

./main -m Meta-Llama-3-8B-Instruct-Q8_0.gguf --temp 0 -i -e -p "<|start_header_id|>system<|end_header_id|>\n\nYou are a helpful assistant.<|eot_id|>" -r "<|eot_id|>" --in-prefix "<|start_header_id|>user<|end_header_id|>\n\n" --in-suffix "<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"

Current behavior:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a helpful assistant.<|eot_id|>
<|eot_id|>
<|start_header_id|>user<|end_header_id|>

Expected behavior:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a helpful assistant.<|eot_id|>
<|start_header_id|>user<|end_header_id|>
@arch-btw
Copy link
Contributor

I can't reproduce the first prompt. Which version of llama.cpp are you using?

@arch-btw
Copy link
Contributor

arch-btw commented May 23, 2024

Also -ins is not used for llama3, it's designed to work with Alpaca model

-ins, --instruct: Run the program in instruction mode, which is specifically designed to work with Alpaca models that excel in completing tasks based on user instructions.

https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md

@Azirine Azirine closed this as completed May 23, 2024
@Azirine Azirine reopened this May 23, 2024
@Azirine Azirine changed the title Llama 3 double <|eot_id|> Llama 3 duplicate <|eot_id|> May 23, 2024
@Azirine Azirine changed the title Llama 3 duplicate <|eot_id|> tokenization: double EOS tokens May 24, 2024
@Azirine Azirine closed this as completed May 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants