CLIPTokenizer

[MatthewWaller]

Hi y'all, I was looking to convert the CLIP Tokenizer found here.

I noticed it seems a lot like the GPT2Tokenizer.swift.

Are there perchance simple adjustments I could make so that it would work for a CLIP model?

Thanks!

[Lukas1h]

Any update on this? I too am interested.

[taydr]

Also interested!

[MatthewWaller]

I have a PR of the tokenizer I used if y'all need it. Take a look here! #30