tiktoken package missing from requirements

[0wwafa]

Some models require tiktoken package for convertion to work.
But titoken package is missing in requirements.txt.

THUDM/codegeex4-all-9b is an example of that
llama.cpp/convert_hf_to_gguf.py will fail saying that tikoken package is needed but not found.

obviously a pip install titoken solves the problem.

[compilade]

This was also discussed when StableLM2 1.6B came out: #5052 (comment)

TL;DR requirements.txt is for required dependencies, not optional ones, but the *-extra Nix devShells do include tiktoken because there's no equivalent to pip install there.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.