Bug: NotImplementedError: BPE pre-tokenizer was not recognized - update get_vocab_base_pre()

[wencan]

What happened?

I tried to use the docker image of llama.cpp to convert the following vector models to gguf format, but I got the same error without exception: NotImplementedError: BPE pre-tokenizer was not recognized - update get_vocab_base_pre()

• sensenova/piccolo-base-zh
• thenlper/gte-base-zh
• BAAI/bge-base-zh-v1.5
• infgrad/stella-base-zh-v2

I understand that this problem may be encountered by all models that use BPE. Please tell me how to solve this problem. It seems that this bug does not exist in older versions of llama.cpp. Which version numbers of the docker image can I try?

Name and Version

ggerganov/llama.cpp full bb70a8fa2901

What operating system are you seeing the problem on?

Linux

Relevant log output

INFO:hf-to-gguf:Loading model: piccolo-base-zh
INFO:gguf.gguf_writer:gguf: This GGUF file is for Little Endian only
INFO:hf-to-gguf:Exporting model...
INFO:hf-to-gguf:gguf: loading model part 'pytorch_model.bin'
INFO:hf-to-gguf:token_embd.weight,               torch.float16 --> F16, shape = {768, 21128}
INFO:hf-to-gguf:position_embd.weight,            torch.float16 --> F32, shape = {768, 512}
INFO:hf-to-gguf:token_types.weight,              torch.float16 --> F32, shape = {768, 2}
INFO:hf-to-gguf:token_embd_norm.weight,          torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:token_embd_norm.bias,            torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.0.attn_q.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.0.attn_q.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.0.attn_k.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.0.attn_k.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.0.attn_v.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.0.attn_v.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.0.attn_output.weight,        torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.0.attn_output.bias,          torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.0.attn_output_norm.weight,   torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.0.attn_output_norm.bias,     torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.0.ffn_up.weight,             torch.float16 --> F16, shape = {768, 3072}
INFO:hf-to-gguf:blk.0.ffn_up.bias,               torch.float16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.0.ffn_down.weight,           torch.float16 --> F16, shape = {3072, 768}
INFO:hf-to-gguf:blk.0.ffn_down.bias,             torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.0.layer_output_norm.weight,  torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.0.layer_output_norm.bias,    torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.1.attn_q.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.1.attn_q.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.1.attn_k.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.1.attn_k.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.1.attn_v.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.1.attn_v.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.1.attn_output.weight,        torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.1.attn_output.bias,          torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.1.attn_output_norm.weight,   torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.1.attn_output_norm.bias,     torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.1.ffn_up.weight,             torch.float16 --> F16, shape = {768, 3072}
INFO:hf-to-gguf:blk.1.ffn_up.bias,               torch.float16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.1.ffn_down.weight,           torch.float16 --> F16, shape = {3072, 768}
INFO:hf-to-gguf:blk.1.ffn_down.bias,             torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.1.layer_output_norm.weight,  torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.1.layer_output_norm.bias,    torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.2.attn_q.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.2.attn_q.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.2.attn_k.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.2.attn_k.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.2.attn_v.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.2.attn_v.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.2.attn_output.weight,        torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.2.attn_output.bias,          torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.2.attn_output_norm.weight,   torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.2.attn_output_norm.bias,     torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.2.ffn_up.weight,             torch.float16 --> F16, shape = {768, 3072}
INFO:hf-to-gguf:blk.2.ffn_up.bias,               torch.float16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.2.ffn_down.weight,           torch.float16 --> F16, shape = {3072, 768}
INFO:hf-to-gguf:blk.2.ffn_down.bias,             torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.2.layer_output_norm.weight,  torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.2.layer_output_norm.bias,    torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.3.attn_q.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.3.attn_q.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.3.attn_k.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.3.attn_k.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.3.attn_v.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.3.attn_v.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.3.attn_output.weight,        torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.3.attn_output.bias,          torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.3.attn_output_norm.weight,   torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.3.attn_output_norm.bias,     torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.3.ffn_up.weight,             torch.float16 --> F16, shape = {768, 3072}
INFO:hf-to-gguf:blk.3.ffn_up.bias,               torch.float16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.3.ffn_down.weight,           torch.float16 --> F16, shape = {3072, 768}
INFO:hf-to-gguf:blk.3.ffn_down.bias,             torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.3.layer_output_norm.weight,  torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.3.layer_output_norm.bias,    torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.4.attn_q.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.4.attn_q.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.4.attn_k.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.4.attn_k.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.4.attn_v.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.4.attn_v.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.4.attn_output.weight,        torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.4.attn_output.bias,          torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.4.attn_output_norm.weight,   torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.4.attn_output_norm.bias,     torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.4.ffn_up.weight,             torch.float16 --> F16, shape = {768, 3072}
INFO:hf-to-gguf:blk.4.ffn_up.bias,               torch.float16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.4.ffn_down.weight,           torch.float16 --> F16, shape = {3072, 768}
INFO:hf-to-gguf:blk.4.ffn_down.bias,             torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.4.layer_output_norm.weight,  torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.4.layer_output_norm.bias,    torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.5.attn_q.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.5.attn_q.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.5.attn_k.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.5.attn_k.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.5.attn_v.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.5.attn_v.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.5.attn_output.weight,        torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.5.attn_output.bias,          torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.5.attn_output_norm.weight,   torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.5.attn_output_norm.bias,     torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.5.ffn_up.weight,             torch.float16 --> F16, shape = {768, 3072}
INFO:hf-to-gguf:blk.5.ffn_up.bias,               torch.float16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.5.ffn_down.weight,           torch.float16 --> F16, shape = {3072, 768}
INFO:hf-to-gguf:blk.5.ffn_down.bias,             torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.5.layer_output_norm.weight,  torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.5.layer_output_norm.bias,    torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.6.attn_q.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.6.attn_q.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.6.attn_k.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.6.attn_k.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.6.attn_v.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.6.attn_v.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.6.attn_output.weight,        torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.6.attn_output.bias,          torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.6.attn_output_norm.weight,   torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.6.attn_output_norm.bias,     torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.6.ffn_up.weight,             torch.float16 --> F16, shape = {768, 3072}
INFO:hf-to-gguf:blk.6.ffn_up.bias,               torch.float16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.6.ffn_down.weight,           torch.float16 --> F16, shape = {3072, 768}
INFO:hf-to-gguf:blk.6.ffn_down.bias,             torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.6.layer_output_norm.weight,  torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.6.layer_output_norm.bias,    torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.7.attn_q.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.7.attn_q.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.7.attn_k.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.7.attn_k.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.7.attn_v.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.7.attn_v.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.7.attn_output.weight,        torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.7.attn_output.bias,          torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.7.attn_output_norm.weight,   torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.7.attn_output_norm.bias,     torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.7.ffn_up.weight,             torch.float16 --> F16, shape = {768, 3072}
INFO:hf-to-gguf:blk.7.ffn_up.bias,               torch.float16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.7.ffn_down.weight,           torch.float16 --> F16, shape = {3072, 768}
INFO:hf-to-gguf:blk.7.ffn_down.bias,             torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.7.layer_output_norm.weight,  torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.7.layer_output_norm.bias,    torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.8.attn_q.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.8.attn_q.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.8.attn_k.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.8.attn_k.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.8.attn_v.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.8.attn_v.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.8.attn_output.weight,        torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.8.attn_output.bias,          torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.8.attn_output_norm.weight,   torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.8.attn_output_norm.bias,     torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.8.ffn_up.weight,             torch.float16 --> F16, shape = {768, 3072}
INFO:hf-to-gguf:blk.8.ffn_up.bias,               torch.float16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.8.ffn_down.weight,           torch.float16 --> F16, shape = {3072, 768}
INFO:hf-to-gguf:blk.8.ffn_down.bias,             torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.8.layer_output_norm.weight,  torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.8.layer_output_norm.bias,    torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.9.attn_q.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.9.attn_q.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.9.attn_k.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.9.attn_k.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.9.attn_v.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.9.attn_v.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.9.attn_output.weight,        torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.9.attn_output.bias,          torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.9.attn_output_norm.weight,   torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.9.attn_output_norm.bias,     torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.9.ffn_up.weight,             torch.float16 --> F16, shape = {768, 3072}
INFO:hf-to-gguf:blk.9.ffn_up.bias,               torch.float16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.9.ffn_down.weight,           torch.float16 --> F16, shape = {3072, 768}
INFO:hf-to-gguf:blk.9.ffn_down.bias,             torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.9.layer_output_norm.weight,  torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.9.layer_output_norm.bias,    torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.10.attn_q.weight,            torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.10.attn_q.bias,              torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.10.attn_k.weight,            torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.10.attn_k.bias,              torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.10.attn_v.weight,            torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.10.attn_v.bias,              torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.10.attn_output.weight,       torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.10.attn_output.bias,         torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.10.attn_output_norm.weight,  torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.10.attn_output_norm.bias,    torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.10.ffn_up.weight,            torch.float16 --> F16, shape = {768, 3072}
INFO:hf-to-gguf:blk.10.ffn_up.bias,              torch.float16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.10.ffn_down.weight,          torch.float16 --> F16, shape = {3072, 768}
INFO:hf-to-gguf:blk.10.ffn_down.bias,            torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.10.layer_output_norm.weight, torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.10.layer_output_norm.bias,   torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.11.attn_q.weight,            torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.11.attn_q.bias,              torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.11.attn_k.weight,            torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.11.attn_k.bias,              torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.11.attn_v.weight,            torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.11.attn_v.bias,              torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.11.attn_output.weight,       torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.11.attn_output.bias,         torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.11.attn_output_norm.weight,  torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.11.attn_output_norm.bias,    torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.11.ffn_up.weight,            torch.float16 --> F16, shape = {768, 3072}
INFO:hf-to-gguf:blk.11.ffn_up.bias,              torch.float16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.11.ffn_down.weight,          torch.float16 --> F16, shape = {3072, 768}
INFO:hf-to-gguf:blk.11.ffn_down.bias,            torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.11.layer_output_norm.weight, torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.11.layer_output_norm.bias,   torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:Set meta model
INFO:hf-to-gguf:Set model parameters
INFO:hf-to-gguf:gguf: context length = 512
INFO:hf-to-gguf:gguf: embedding length = 768
INFO:hf-to-gguf:gguf: feed forward length = 3072
INFO:hf-to-gguf:gguf: head count = 12
INFO:hf-to-gguf:gguf: layer norm epsilon = 1e-12
INFO:hf-to-gguf:gguf: file type = 1
INFO:hf-to-gguf:Set model tokenizer
WARNING:hf-to-gguf:

WARNING:hf-to-gguf:**************************************************************************************
WARNING:hf-to-gguf:** WARNING: The BPE pre-tokenizer was not recognized!
WARNING:hf-to-gguf:**          There are 2 possible reasons for this:
WARNING:hf-to-gguf:**          - the model has not been added to convert_hf_to_gguf_update.py yet
WARNING:hf-to-gguf:**          - the pre-tokenization config has changed upstream
WARNING:hf-to-gguf:**          Check your model files and convert_hf_to_gguf_update.py and update them accordingly.
WARNING:hf-to-gguf:** ref:     https://github.com/ggerganov/llama.cpp/pull/6920
WARNING:hf-to-gguf:**
WARNING:hf-to-gguf:** chkhsh:  8e62295832751ca1e8f92f2226f403dea30dc5165e448b5bfa05af5340c64ec7
WARNING:hf-to-gguf:**************************************************************************************
WARNING:hf-to-gguf:

Traceback (most recent call last):
  File "/app/./convert_hf_to_gguf.py", line 3688, in <module>
    main()
  File "/app/./convert_hf_to_gguf.py", line 3682, in main
    model_instance.write()
  File "/app/./convert_hf_to_gguf.py", line 401, in write
    self.prepare_metadata(vocab_only=False)
  File "/app/./convert_hf_to_gguf.py", line 394, in prepare_metadata
    self.set_vocab()
  File "/app/./convert_hf_to_gguf.py", line 2414, in set_vocab
    tokens, toktypes, tokpre = self.get_vocab_base()
  File "/app/./convert_hf_to_gguf.py", line 482, in get_vocab_base
    tokpre = self.get_vocab_base_pre(tokenizer)
  File "/app/./convert_hf_to_gguf.py", line 620, in get_vocab_base_pre
    raise NotImplementedError("BPE pre-tokenizer was not recognized - update get_vocab_base_pre()")
NotImplementedError: BPE pre-tokenizer was not recognized - update get_vocab_base_pre()

[wencan]

I compiled the llama.cpp file with version b2636, successfully converted sensenova/piccolo-base-zh to gguf format, and quantized it to Q4_K-M format.

[NeuralAIM]

Same with sentence-transformers/LaBSE

b2636 - working

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[azraimahadan]

This is still an issue today.

[SinpackKonmakan]

I wonder if this issue solved

[ripani-openimpact]

Still an issue with Lora and Phi4

"""
WARNING:hf-to-gguf:

WARNING:hf-to-gguf:**************************************************************************************
WARNING:hf-to-gguf:** WARNING: The BPE pre-tokenizer was not recognized!
WARNING:hf-to-gguf:** There are 2 possible reasons for this:
WARNING:hf-to-gguf:** - the model has not been added to convert_hf_to_gguf_update.py yet
WARNING:hf-to-gguf:** - the pre-tokenization config has changed upstream
WARNING:hf-to-gguf:** Check your model files and convert_hf_to_gguf_update.py and update them accordingly.
WARNING:hf-to-gguf:** ref: #6920
WARNING:hf-to-gguf:**
WARNING:hf-to-gguf:** chkhsh: 53e325976a6e142379c19b09afcae354f2f496f147afa8f9e189a33fe4e3024e
WARNING:hf-to-gguf:**************************************************************************************
WARNING:hf-to-gguf:

Traceback (most recent call last):
...
raise FileNotFoundError(f"File not found: {tokenizer_path}")
FileNotFoundError: File not found: saved_model/tokenizer.model

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
....
FileNotFoundError: [Errno 2] No such file or directory: 'saved_model/tokenizer.json'

During handling of the above exception, another exception occurred:

....

raise NotImplementedError("BPE pre-tokenizer was not recognized - update get_vocab_base_pre()")

NotImplementedError: BPE pre-tokenizer was not recognized - update get_vocab_base_pre()

[zzwtop1]

I wonder if this issue solved.model：DeepSeek-R1-0528-Qwen3-8B

[zzwtop1]

找到了解决办法，运行convert_hf_to_gguf_update.py即可，但是需要先修改convert_hf_to_gguf_update.py文件的内容，models =[ {"name": "llama-spm", "tokt": TOKENIZER_TYPE.SPM, "repo": "https://huggingface.co/meta-llama/Llama-2-7b-hf", }...]之中加上{"name": "deepseek-r1-qwen3", "tokt": TOKENIZER_TYPE.BPE, "repo": "https://huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B", }，此外你需要确保models =[ {"name": "llama-spm", "tokt": TOKENIZER_TYPE.SPM, "repo": "https://huggingface.co/meta-llama/Llama-2-7b-hf", }...]里面对应的huggingface模型你有权限下载，否则convert_hf_to_gguf_update.py会运行失败，如果模型拥有者不同意你的申请，可以先删除掉再运行convert_hf_to_gguf_update.py

[visitsb]

Check function convert_hf_to_gguf -> get_vocab_base_pre().

There is a a sample text which is tokenized using the tokenizer and the checksum is compared to a known list of tokenizers.
If the checksum does not match to any known checksums, you get the error - "BPE pre-tokenizer was not recognized - update get_vocab_base_pre()".

This error occurs when the tokenizer vocabulary for an existing tokenizer has been modified.

In any case, whenever new tokens are added to change the tokenizer vocabulary, check if the checksum value for check function remains unchanged. If that changes, that means your changes affect the existing tokenization behavior. Using the modified tokenizer with a model will only give skewed and incorrect results which is never intended for fine-tuning, but only when you are training from scratch.

For e.g. if you use HuggingFace tokenizer's train_new_from_iterator(), add_tokens() to add new vocabulary to an existing tokenizer, then it's preferable to use AddedToken with single_word=True to avoid affecting existing vocabulary.