Option to suppress tokens

[pprobst]

Hello. As far as I know, whispercpp does not support the option to suppress particular tokens / numerals during inference, like WhisperX does. This is particularly useful, for example, if we want to transcribe numbers literally, e.g., "one" instead of "1".

Is there any interest in adding support for this feature?

[ggerganov]

Have you tried the grammar functionality: #1229 - could be useful

[pprobst]

No, I didn't know it was a thing. I will look into it. But if I understood it correctly from a brief reading, this is awesome; given a context-specific corpus, we can use it to "boost the weight" of certain words. There should be an example or something in the README to give more attention to this feature because it's a game-changer 👀

[pprobst]

Oh! I just found exactly what I needed in the source code (the grammar functionality will still be handy for me, though, once I get it running for my own needs).

https://github.com/ggerganov/whisper.cpp/blob/2623640cd6b8e077ffb1fbe3c265208da3a0ca6d/whisper.cpp#L4473-L4478

And

https://github.com/ggerganov/whisper.cpp/blob/2623640cd6b8e077ffb1fbe3c265208da3a0ca6d/whisper.cpp#L4567-L4575

Very handy. Just need to add the tokens I won't use. I'm closing this issue now.

[flatsiedatsie]

Would it be possible to load such lists from an external file instead of having to recompile?

[josharian]

Seconded. I have the same use case (suppressing 0123456789%$£, as in whisperx). It'd be nice to have this available in vanilla whisper.cpp via the command line. If there's openness, I'm game to try my hand at a PR. The main question is what the flags should be. (Should it be a simple --suppress-digits, like whisperx? Or take a string, which we split into tokens? Etc.)

[josharian]

Hmmm. It's more than what @pprobst found, because there are lots of all-numeric tokens, such as "500" (which is not just "5", "0", "0"). Grammars might indeed be a better choice here...

[pprobst]

Hmmm. It's more than what @pprobst found, because there are lots of all-numeric tokens, such as "500" (which is not just "5", "0", "0"). Grammars might indeed be a better choice here...

Yeah. I noticed that afterward. Then, I got the list of tokens (IDs) that whisperx returned to me when I used the suppress_numerals options and hardcoded them into whispercpp. Ugly but it worked for me. Would be cool if there was a similar option in whispercpp.

[josharian]

While I work on grammars, here's a quick patch folks can apply as desired:

        static const std::string numbery = "0123456789%$£";
        for (int i = 0; i < vocab.token_beg; i++) {
            const std::string & token = vocab.id_to_token.at(i);
            if (token.find_first_of(numbery) != std::string::npos) {
                logits[i] = -INFINITY;
            }
        }

(This goes just after "suppress non-speech tokens".)

It appears to cost about 5% runtime, not inconsiderable.

If we wanted to add proper support for a simple "omit these ascii characters", without grammars, this could be made much cheaper by doing the find_first_of calcs once, not every time.

[wttdotm]

@josharian I just want to say thank you for this simple and goated patch. I was having a nightmare of a time dealing with word-level timestamps for dollar amounts (eg. "$45" being three different words but being treated as one by whisper) and after hours and hours of banging my head against unhelpful searches and repos this works perfectly.

[josharian]

Thanks. :)

If you want to get rid of the performance penalty for that patch, you can try pulling in this still-pretty-small commit, which I just rebased onto master: josharian@c664398. (It worked months ago when I wrote it, but I haven't tested it since.)

[ThioJoe]

Ok so i found out there's a --suppress-regex argument. Which can be used without needing to custom compile anything.

I wanted to not just match specific tokens like all the numbers and specific punctuation, but rather wanted to make regex that would match anything except tokens that only contain basic characters and certain allowed punctuation. Basically whitelist certain characters.

And long story short that wasn't as simple as I thought because the regex will also match special tokens which look like [_EOT_] and others.

But I was finally able to figure it out. Adding this to the command should work:

--suppress-regex "^(?!^\[_(EOT|SOT|TRANSLATE|TRANSCRIBE|SOLM|PREV|NOSP|NOT|BEG|TT)_\d*\]$).*[^a-zA-Z\-?.'\x22\x20\x2C].*"

Anything in this part between the [^ and ] are characters that are allowed:

[^a-zA-Z\-?.'\x22\x20\x2C]

The ^ character basically says "don't match anything in this group". a-zA-Z means any upper or lower case characters.
The \x22\x20\x2C are just hex representations of the characters for double quote, space, and comma, because including them literally was screwing with the argument parsing. The other characters like the question mark and single quote and period are literal. The backslash in front of the - is an escape character though so it means include literal hyphen.

All the stuff before is basically to not match any of the special tokens.

If you want to put it into a batch file, you need to escape any ^ characters with another ^ character, and also escape any ! character with a ^ too.
So within a batch file it would look like:

--suppress-regex "^^(?^!^^\[_(EOT|SOT|TRANSLATE|TRANSCRIBE|SOLM|PREV|NOSP|NOT|BEG|TT)_\d*\]$).*[^^a-zA-Z\-?.'\x22\x20\x2C].*"

---

Also here's another related discussion from the whisper repo: #1041
And the commit which added the regex, and referenced that discussion: c8eeb93

---

Note, I think there is what I'd consider a bug in the token comparison when using regex.

For example, take this simpler regex:

--suppress-regex ".*[0123456789].*"

Where Anything between .*[ and ].* and suppressed characters. It should only suppress tokens with literal numbers in it. But because special tokens are included in those being compared for suppression, it screws with the timestamp special tokens which look like [_TT_999] or whatever other number for example instead of 999. You can see them if you do --print-special on a normal run.

I'll probably make a separate issue report for that. I don't think the regex should match special tokens, unless perhaps there's some other argument switch to explicitly do so.

My big long regex above does work around that problem though.