add support of Aquila 7B models

[ftgreat]

We released and open-sourced Aquila 7B series , including AquilaChat-7B(https://github.com/FlagAI-Open/FlagAI/tree/master/examples/Aquila/Aquila-chat) and Aquila-7B(https://github.com/FlagAI-Open/FlagAI/tree/master/examples/Aquila/Aquila-pretrain), which support both Chinese and English knowledge.

The model architectures are almost same as LLaMa, except one GPT2-like BPE tokenizer is used.

Could llama.cpp repo add our Aquila 7B models and how to adapt for the BPE tokenizer? Thanks very much.

[howard0su]

llama.cpp implemented llama_tokenizer as a separate class. You can implement GPT2Tokenizer similar to that.

The interface today doesn't support different tokenizer, however it should be relative easy to fix by introducing a new setting.

[ftgreat]

#2228

I add one pr for support bpe tokenizer in convert.py.
@howard0su could you please review it,thanks.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.