You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
just realized that seemingly some recent changes make the script break on creating the llama-spm contents. it runs through without that line. which is my quick and lazy workaround atm (also in a quickly hacked kaggle script to run through the steps to fix the pre tokenizer issue). sorry i cannot look into this further, and maybe it is just some intermediate inconsistency that gets solved in the process of the current edits in the repo. or maybe you want to look into it.