convert : fix byte tokens for --vocab-type hfft

[Artefact2]

This is inspired by 9f297f8, which got lost during the refactoring in 6efb8eb.

Fixes #5064.

[Artefact2]

Can I get a ack/nack on this?

[felladrin]

I've checked out your branch, compiled main, and tested it. It solved the issue for me 🙏

Command:

llama.cpp/main -m model/ggml-model-f32.gguf -p "<|im_start|>user\nHow do I incorporate visual elements into my writing?<|im_end|>\n<|im_start|>assistant" -n 250 -e --repeat_penalty 1.0 -c 0 --temp 0.1 --min-p 0.1

Output (without the changes from this PR):

<|im_start|>user<0x0A>How do I incorporate visual elements into my writing?<|im_end|><0x0A><|im_start|>assistant<0x0A>I do not have access to the specific details of the text material. However, here are some examples of how to use the following:<0x0A><0x0A>

Output (with the changes from this PR):

<|im_start|>user
How do I incorporate visual elements into my writing?<|im_end|>
<|im_start|>assistant
I do not have access to the specific details of the text material. However, I can provide you with a list of the following:

So I hope it can be merged soon, as it will finally allow us to correctly convert Llama/Mistral models that doesn't have the tokenizer.model file along with the model.

[x4080]

waiting for this too

[slaren]

Is there any reason to not implement this logic directly in get_token_type? It seems out of place here.

[ggerganov]

@Artefact2 Sorry for the delay - I'm somehow just seeing the PR

I see this is now closed - are you considering a new PR?

[Artefact2]

No, the bug is fixed as far as I'm concerned. If my patch is not up to the standard required, someone else can take care of it.

[maziyarpanahi]

the bug is fixed

Thanks @Artefact2 for your contribution. Was it fixed by other PRs already merged into the main, or it was fixed for you and you moved on? (totally understand if it's the latter)

[Artefact2]

Was it fixed by other PRs already merged into the main

Yes! See #5341.

[maziyarpanahi]

Was it fixed by other PRs already merged into the main

Yes! See #5341.

Many thanks!