Skip to content

Inference sometimes gens control chars such as ASCII 28 (\xc1) which can pollute context / cause havoc #2758

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ProjectAtlantis-dev opened this issue Aug 24, 2023 · 3 comments

Comments

@ProjectAtlantis-dev
Copy link

ProjectAtlantis-dev commented Aug 24, 2023

While troubleshooting #2678, I caught llama2 (or something?) generating a bunch of non-printable characters (ASCII 28 specifically) in my bot conversations. It is possible that this is the root cause of #2678

Since I was feeding LLM responses into a transcript in the context, it seems then these pollute the conversation (server.cpp does not directly write responses into the context like how main.cpp does). This would cause my next round of token processing to go crazy and spin the GPU but not return anything (presumably trying to do something w the control characters)

I was using upstage-llama-2-70b-instruct-v2.ggmlv3.q5_K_M.bin on Metal M1 64GB. I am not using MPS

This was observed with build bf83bff

I have not been able to reliably repro this yet with dadbed9 so take it with a grain of salt, but if anyone else is suffering from #2678 check to see if the data returned contains non-printables. I'm not sure if these are coming from code or data but how those got the 70b data in the first place is probably another question

Example here
#2678 (comment)

@ProjectAtlantis-dev ProjectAtlantis-dev changed the title llama2 sometimes generates non-printables such as ASCII 28 (\xc1) which can contaminate context for subsequent processing llama sometimes generates non-printables such as ASCII 28 (\xc1) which can contaminate context for subsequent processing Aug 24, 2023
@ProjectAtlantis-dev ProjectAtlantis-dev changed the title llama sometimes generates non-printables such as ASCII 28 (\xc1) which can contaminate context for subsequent processing llama sometimes gens control chars such as ASCII 28 (\xc1) which can pollute context Aug 24, 2023
@ProjectAtlantis-dev ProjectAtlantis-dev changed the title llama sometimes gens control chars such as ASCII 28 (\xc1) which can pollute context Inference sometimes gens control chars such as ASCII 28 (\xc1) which can pollute context Aug 24, 2023
@ProjectAtlantis-dev ProjectAtlantis-dev changed the title Inference sometimes gens control chars such as ASCII 28 (\xc1) which can pollute context Inference sometimes gens control chars such as ASCII 28 (\xc1) which can pollute context / cause havoc Aug 24, 2023
@lshzh-ww
Copy link
Contributor

Hi @ProjectAtlantis-dev ,

Could you attempt to backport #2699 to your code? I assume you are not using the latest llama.cpp code and are missing the correct fix for the problem you reported.

@ProjectAtlantis-dev
Copy link
Author

ProjectAtlantis-dev commented Aug 24, 2023

I am testing with dadbed9 and have not gotten any more control characters.

In any case, it is a pre-GGUF model so perhaps has been overcome by events

@ProjectAtlantis-dev
Copy link
Author

Have not seen any more control chars and since pre-GGUF model I am closing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants