-
Notifications
You must be signed in to change notification settings - Fork 12k
Inference sometimes gens control chars such as ASCII 28 (\xc1) which can pollute context / cause havoc #2758
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Hi @ProjectAtlantis-dev , Could you attempt to backport #2699 to your code? I assume you are not using the latest llama.cpp code and are missing the correct fix for the problem you reported. |
I am testing with dadbed9 and have not gotten any more control characters. In any case, it is a pre-GGUF model so perhaps has been overcome by events |
Have not seen any more control chars and since pre-GGUF model I am closing |
Uh oh!
There was an error while loading. Please reload this page.
While troubleshooting #2678, I caught llama2 (or something?) generating a bunch of non-printable characters (ASCII 28 specifically) in my bot conversations. It is possible that this is the root cause of #2678
Since I was feeding LLM responses into a transcript in the context, it seems then these pollute the conversation (server.cpp does not directly write responses into the context like how main.cpp does). This would cause my next round of token processing to go crazy and spin the GPU but not return anything (presumably trying to do something w the control characters)
I was using upstage-llama-2-70b-instruct-v2.ggmlv3.q5_K_M.bin on Metal M1 64GB. I am not using MPS
This was observed with build bf83bff
I have not been able to reliably repro this yet with dadbed9 so take it with a grain of salt, but if anyone else is suffering from #2678 check to see if the data returned contains non-printables. I'm not sure if these are coming from code or data but how those got the 70b data in the first place is probably another question
Example here
#2678 (comment)
The text was updated successfully, but these errors were encountered: