Add BPE pre-tokenization for Command-R.#7033

dranger003 wants to merge 3 commits intoggml-org:masterggml-org/llama.cpp:masterfrom

dranger003:bpe-pretok-command-rdranger003/llama.cpp:bpe-pretok-command-rCopy head branch name to clipboard