Unexpected end of JSON input

[thiner]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

Streaming output should work correctly.

Current Behavior

Streaming output is interrupted with below error message:

{
  "error": true,
  "message": "Unexpected end of JSON input"
}

The same API is working properly with same request body if call with Postman, the only difference is header set to Accept: application/json in Postman.

Environment and Context

Please provide detailed information about your computer setup. This is important in case the issue is not reproducible except for under certain specific conditions.

Running in Docker image, the image information:

• Base image: nvidia/12.1.0-devel-ubuntu22.04
• Python: 3.10.2
• Installed llama-cpp-python in Dockerfile by CMAKE_ARGS="-DLLAMA_CUBLAS=on" pip3 install llama-cpp-python[server]

Failure Information (for bugs)

The Web UI shows error message:

{
  "error": true,
  "message": "Unexpected end of JSON input"
}

Steps to Reproduce

Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.

Just call the v1/chat/completions API, and set the header value Accept: text/event-stream.

Note: Many issues seem to be regarding functional or performance issues / differences with llama.cpp. In these cases we need to confirm that you're comparing against the version of llama.cpp that was built with your python package, and which parameters you're passing to the context.

Try the following:

1. git clone https://github.com/abetlen/llama-cpp-python
2. cd llama-cpp-python
3. rm -rf _skbuild/ # delete any old builds
4. python -m pip install .
5. cd ./vendor/llama.cpp
6. Follow llama.cpp's instructions to cmake llama.cpp
7. Run llama.cpp's ./main with the same arguments you previously passed to llama-cpp-python and see if you can reproduce the issue. If you can, log an issue with llama.cpp

Failure Logs

There is not any error message output by llama-cpp-python.

llama_print_timings:        load time =    4354.71 ms

 llama_print_timings:      sample time =      14.60 ms /    62 runs   (    0.24 ms per token,  4245.70 tokens per second)

 llama_print_timings: prompt eval time =     189.30 ms /     7 tokens (   27.04 ms per token,    36.98 tokens per second)

 llama_print_timings:        eval time =    2788.22 ms /    61 runs   (   45.71 ms per token,    21.88 tokens per second)

 llama_print_timings:       total time =    3134.24 ms

 Llama.generate: prefix-match hit

 

 llama_print_timings:        load time =    4354.71 ms

 llama_print_timings:      sample time =       1.86 ms /     8 runs   (    0.23 ms per token,  4294.15 tokens per second)

 llama_print_timings: prompt eval time =    1138.13 ms /   128 tokens (    8.89 ms per token,   112.47 tokens per second)

 llama_print_timings:        eval time =     375.45 ms /     7 runs   (   53.64 ms per token,    18.64 tokens per second)

 llama_print_timings:       total time =    1774.18 ms

 INFO:     10.70.0.104:47466 - "POST /v1/chat/completions?path=v1&path=chat&path=completions HTTP/1.1" 200 OK

 Llama.generate: prefix-match hit

 INFO:     10.70.0.200:16312 - "POST /v1/chat/completions?path=v1&path=chat&path=completions HTTP/1.1" 200 OK

Please include any relevant log snippets or files. If it works under one configuration but not under another, please provide logs for both configurations and their corresponding outputs so it is easy to see where behavior changes.

Also, please try to avoid using screenshots if at all possible. Instead, copy/paste the console output and use Github's markdown to cleanly format your logs for easy readability.

[aniljava]

This is the limitation of the FastAPI. By default it expects, Accept: application/json.

#1088 addresses.

OpenAI support the text/streaming as Accept header as well as response is marked same. But its openapi.yaml specs or documentation does not explicitly call out text/streaming.

[thiner]

Thanks so much for your quick patching. I will check-out your code changes and verify locally.

[thiner]

I tried latest code, seems the issue still existing. Even worse, the output becomes all "#######".

I tried with model TheBloke/openbuddy-mixtral-7bx8-v16.3-32k-GGUF
and the arguments as below:

command:
            - python3
            - '-m'
            - llama_cpp.server
          args:
            - '--host'
            - 0.0.0.0
            - '--port'
            - '8000'
            - '--model'
            - /opt/models/localai/openbuddy-mixtral-7bx8-v16.3-32k.Q5_K_M.gguf
            - '--model_alias'
            - gpt-3.5-turbo
            - '--chat_format'
            - openbuddy
            - '--api_key'
            - $(API-KEY)
            - '--tensor_split'
            - '0.5'
            - '0.5'
            - '--n_gpu_layers'
            - '-1'
            - '--n_threads'
            - '16'
            - '--interrupt_requests'
            - 'false'
            - '--n_ctx'
            - '32500'

[aniljava]

@thiner ,
The ###... is related to offload_kqv related bug in llama.cpp.
#1089 (comment)

pass, --offload_kqv true with the server command, offload_kqv = True with the Llama init.

[thiner]

I tested with the latest version v0.2.30, most of the streaming output can complete normally. But it still appearing the Unexpected end of JSON... error occasionally. I encountered the error once this morning in roughly 30 calls. Anyway, I think it's workable for most of time. Thank you very much for the quick patching.

[abetlen]

@thiner do you know what the finish reason is? I believe grammar sampling should suppress the eos token but this may not be the case and it terminates before enclosing the end of the input.

[thiner]

I don't know, I couldn't find any clue from the logs. It just didn't output the timing information, as I pasted at the beginning of this thread.

[thiner]

Unfortunately, v0.2.38 encountered this problem again. I tried several mixtral models, all responses are interrupted after maybe 100~200 words.