The prompt processing takes ages now

[BadisG]

Hello,
Now that I've updated llama.cpp to lama_cpp_python-0.2.14, I've noticed that when a new prompt is ready to be processed, it takes a lot of time (usually ~50 tokens context takes less than 2 seconds, now it takes 40 seconds!), the following outputs (with the same prompts) are ok though.

I also noticed that when I start the output just after "ASSISTANT:" (for Xwin), it just breaks the output, it never happened before

[Kenshiro-28]

I also have problems with v0.2.14. I'm using dolphin-2.2.1-mistral-7B, the model has weird or blank responses, it was working great in v0.2.13.

[abetlen]

@BadisG can you update to the latest version and try running the example from README and lmk if the issue appears there as well.

>>> from llama_cpp import Llama
>>> llm = Llama(model_path="./models/7B/llama-model.gguf", n_gpu_layers=-1)
>>> output = llm("Q: Name the planets in the solar system? A: ", max_tokens=32, stop=["Q:", "\n"], echo=True)
>>> print(output)

[BadisG]

@abetlen I got this

llama_new_context_with_model: compute buffer total size = 81.63 MB
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 0 | VSX = 0 |

llama_print_timings:        load time =    2546.26 ms
llama_print_timings:      sample time =       0.79 ms /     5 runs   (    0.16 ms per token,  6353.24 tokens per second)
llama_print_timings: prompt eval time =    2546.21 ms /    15 tokens (  169.75 ms per token,     5.89 tokens per second)
llama_print_timings:        eval time =    1693.41 ms /     4 runs   (  423.35 ms per token,     2.36 tokens per second)
llama_print_timings:       total time =    4250.76 ms
{'id': 'cmpl-ca3b1e30-5219-4739-86de-50b57816c456', 'object': 'text_completion', 'created': 1699641445, 'model': './models/7B/llama-model.gguf', 'choices': [{'text': 'Q: Name the planets in the solar system? A: 1. Mercury', 'index': 0, 'logprobs': None, 'finish_reason': 'stop'}], 'usage': {'prompt_tokens': 15, 'completion_tokens': 5, 'total_tokens': 20}}

[abetlen]

So it looks like the issue may be in llama.cpp because the prompt eval time is consistent with the slowdown you reported. Do you mind testing llama.cpp standalone?

[janpawellek]

I also notice slow prompt generation 🤔
In the output I get, I have BLAS = 0 even though I installed llama-cpp-python with CMAKE_ARGS="-DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS" FORCE_CMAKE=1 pip install llama-cpp-python --force-reinstall --upgrade --no-cache-dir.

@BadisG, in your output it also says BLAS = 0. Might this be the issue?

[BadisG]

@janpawellek No, look at the very first post, I have BLAS = 1 there.

[titos-carrasco]

Same problem here with llama-cpp-python 0.2.19. But only in the first call

I'm using a first prompt with about 1600 tokens, The rrocessing time is about 200.000 ms. After that I concatenate all the prompts and send it to LLM. The processing time now is ok (I'm using cpu only)

[abetlen]

@titos-carrasco that's expected, prompt processing on the CPU is slower than you usually see with a GPU. The reason that subsequent calls are faster is because the Llama class keeps track of processed tokens so when you send a concatenated string only the new tokens at the end need to be processed.

[titos-carrasco]

Interesting, if each new text is identical to one previously entered, it calls for the reuse of the evaluation of the prompt done before. In fact, if the same prompt is entered with a new word, that prompt is reevaluated.

How can save the evaluation of the prompt to reuse it?

Sorry for my english, i'm using chatgpt to translate from spanish to english

[abetlen]

@titos-carrasco no problem and sorry for the late reply, due to the transformers auto-regressive architecture it's actually only possible to re-use tokens from the prefix / start of the sequence, this is because attention is applied to previously seen tokens but not future tokens.

For example you can add a new word at the end of a sentence and only the new word is processed however if you add a new word at the beginning of the sentence the entire sentence has to be processed.

[abetlen]

@BadisG any luck with the latest version?

[BadisG]

@abetlen Yeah it's working fine with the new versions now, thank you for the fix!