Inference sometimes gens control chars such as ASCII 28 (\xc1) which can pollute context / cause havoc

[ProjectAtlantis-dev]

While troubleshooting #2678, I caught llama2 (or something?) generating a bunch of non-printable characters (ASCII 28 specifically) in my bot conversations. It is possible that this is the root cause of #2678

Since I was feeding LLM responses into a transcript in the context, it seems then these pollute the conversation (server.cpp does not directly write responses into the context like how main.cpp does). This would cause my next round of token processing to go crazy and spin the GPU but not return anything (presumably trying to do something w the control characters)

I was using upstage-llama-2-70b-instruct-v2.ggmlv3.q5_K_M.bin on Metal M1 64GB. I am not using MPS

This was observed with build bf83bff

I have not been able to reliably repro this yet with dadbed9 so take it with a grain of salt, but if anyone else is suffering from #2678 check to see if the data returned contains non-printables. I'm not sure if these are coming from code or data but how those got the 70b data in the first place is probably another question

Example here
#2678 (comment)

[lshzh-ww]

Hi @ProjectAtlantis-dev ,

Could you attempt to backport #2699 to your code? I assume you are not using the latest llama.cpp code and are missing the correct fix for the problem you reported.

[ProjectAtlantis-dev]

I am testing with dadbed9 and have not gotten any more control characters.

In any case, it is a pre-GGUF model so perhaps has been overcome by events

[ProjectAtlantis-dev]

Have not seen any more control chars and since pre-GGUF model I am closing