Usage of Ġ in BPE tokenizer

[maschasap]

Hello,
I want to add new words to my BPE tokenizer. I know the symbol Ġ means the end of a new token and the majority of tokens in vocabs of pre-trained tokenizers start with Ġ. Assume I want to add the word Salah to my tokenizer. I tried to add both Salah token and ĠSalah:
tokenizer.add_tokens(['Salah', 'ĠSalah']) # they get 50265 and 50266 values respectively.
However, when I tokenize a sentence where Salah appears, the tokenizer will never return me the second number (neither when using .tokenize nor .encode), for instance:
tokenizer.tokenize('I love Salah and salad') returns ['I', 'Ġlove', 'Salah', 'Ġand', 'Ġsalad'].
The question is: should I use the symbol Ġ when adding new tokens or the tokenizer does it itself? Or, probably, it must be specified manually?
Thanks in advance!

[patil-suraj]

@maschasap
AFAIK, you won't need to add Ġ when adding a new token.
And you can use the convert_tokens_to_string method to convert these tokens to their respective strings.

tagging @mfuntowicz for more info

[maschasap]

@patil-suraj thanks for your response! But still why doesn't the tokenizer add Ġ symbol when returning the tokenized sentence in the example? Does it mean it still needs to be tuned or is it OK?

[maschasap]

@patil-suraj btw using the method convert_tokens_to_string the whitespace between the words love and Salah really disappears :(
tokenizer.convert_tokens_to_string(tokenizer.tokenize('I love Salah and salad')) outputs 'I loveSalah and salad

[maschasap]

@LysandreJik @mfuntowicz may I ask you for help?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jlanday]

Can someone help me understand the symbol Ġ ? I am running into the same thing , want to make sure I am not breaking things down the line

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Focus]

There is also a difference here between the fast and slow tokenisers:

import transformers
tokeniser1 = transformers.RobertaTokenizer.from_pretrained('roberta-base')
tokeniser2 = transformers.RobertaTokenizerFast.from_pretrained('roberta-base')
tokeniser1.add_tokens(['Salah'])
tokeniser2.add_tokens(['Salah'])
print(tokeniser1.tokenize('I love Salah and salad'))
print(tokeniser2.tokenize('I love Salah and salad'))

Outputs:

['I', 'Ġlove', 'Salah', 'and', 'Ġsalad']
['I', 'Ġlove', 'Ġ', 'Salah', 'Ġand', 'Ġsalad']

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.