tests : add test-tokenizer-0.sh + fix some tokenizers (ggml-org#7036)

* tests : add test-tokenizer-0.sh

* unicode : add all unicode number ranges

* starcoder : fix pre-tokenizer

* tests : add test that fails with DeepSeek tokenizers

* falcon : fix regex

* unicode : regenerate unicode tables

* refact : add tokenizer model

* lint : fix

* tests : disable failing tests

ggml-ci

* refact : add tests files

ggml-ci

* convert : print -> logging

ggml-ci

* lint : fix

* unicode : digit -> number

* phi-3 : update

Author: ggerganov

tests/test-tokenizer-0.py

@@ -1,5 +1,5 @@
-import time
 import argparse
+import time
 
 from transformers import AutoTokenizer