Two tokenizer initialization methods result in inconsistent segmentation results for special words

[KinvenW]

System Info

transformers==4.17.0
torch==1.10.0
python==3.7.3

Who can help?

@ArthurZucker

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

# xlm-roberta-base directory: git clone https://huggingface.co/xlm-roberta-base
from transformers import XLMRobertaTokenizer
tokenizer_a = XLMRobertaTokenizer.from_pretrained('xlm-roberta-base/')
tokenizer_b = XLMRobertaTokenizer('xlm-roberta-base/sentencepiece.bpe.model')

t = 'texta<s>textb'
print(tokenizer_a.tokenize(t))
print(tokenizer_b.tokenize(t))

Expected behavior

# what I expect is that both outputs: 
['▁text', 'a', '<s>', '▁text', 'b']
['▁text', 'a', '<s>', '▁text', 'b']

# However, in reality, their outputs are as follows:
['▁text', 'a', '<s>', '▁text', 'b']
['▁text', 'a', '<', 's', '>', 'text', 'b']

Why these two tokenizers have different segmentation results for special words?

[ArthurZucker]

Hey! Which version of transformers are you using? I tried to reproduce this but it did not work. Download the sentencepiece model from the official repo, and it worked on main~

[KinvenW]

Hey! Which version of transformers are you using? I tried to reproduce this but it did not work. Download the sentencepiece model from the official repo, and it worked on main~

my transformers version is 4.17.0.

[ArthurZucker]

Ok, I can confirm that this is not working for me either! Thanks for reporting. This is most probably a problem related to add_tokens in the init:

>>> tokenizer_a.unique_no_split_tokens
['</s>', '<mask>', '<pad>', '<s>', '<unk>']
>>> tokenizer_b.unique_no_split_tokens

[ArthurZucker]

Linking this with #23909 as the core problem is similar:

• Adding a special token does not recreate the Trie (when the special tokens are initialized)
• from_pretrained calls added_tokens = tokenizer.sanitize_special_tokens() which is when the special tokens are added to unique_no_split
• Initialising a model from a sentencepiece vocab file does not initialize the Trie (used to split the tokens), since the special tokens are not sanitized.

[ArthurZucker]

I've had to take on a few model addition, the fix PR will be updated soon! It requires a bit more refactoring than initially thought!

[ArthurZucker]

Once transformers is released, I'll merge the fix!