KV_cache not offloading to GPU. Slows down the process a lot!

[anuragrawal2024]

Hi,

I think a recent change might have caused this. I am using llama-2-7b-chat.Q4_K_M.gguf for a local Q&A RAG pipeline, created using LlamaIndex. I developed a proof of concept on a machine using 0.2.13 version and saw this in the output:

llama_kv_cache_init: offloading v cache to GPU
llama_kv_cache_init: offloading k cache to GPU
llama_kv_cache_init: VRAM kv self = 2048.00 MB

When I recently installed llama-cpp-python on a new machine, I don't see this in output anymore and my process has slowed down significantly. Can you please advise? Let me know if you need anything additional.

[konrad0101]

I get the same issue with llama-index and llama-cpp-python 0.2.22. But version 0.2.20 works fine. My install command is:

CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install llama-cpp-python==0.2.20 --force-reinstall --upgrade --no-cache-dir --verbose

The slow down is substantial, going from 12ms/token to about 113ms/token using a 3090 GPU and a 7B model. The solution for me was to revert to 0.2.20 of llama-cpp-python.

The difference in the output for me is that version 0.2.20 has these lines that are otherwise missing in 0.2.22:

llama_kv_cache_init: offloading v cache to GPU
llama_kv_cache_init: offloading k cache to GPU
llama_kv_cache_init: VRAM kv self = 487.50 MiB

The offloading to GPU is also slightly different:

0.2.20:

llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 35/35 layers to GPU

0.2.22:

llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
....
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU

I think the llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output line might be a hint for what is wrong.

[anuragrawal2024]

Yes, I noticed all those differences between 0.2.13 and the latest version 0.2.22. I went back to 0.2.13 because I can't afford the slowdown.

[oobabooga]

A fix has been posted here: https://www.reddit.com/r/Oobabooga/comments/18ifjxb/comment/kdgmr2v/?utm_source=share&utm_medium=web2x&context=3

cc @abetlen

[abetlen]

@utility-aagrawal should be fixed in the new release with the offload_kqv=True option

[abetlen]

I'm going to close this for now assuming the issue was related to the kv cache offloading, will reopen if necessary.

[cognitivetech]

the issue may be closed, but the problem persists... I have taken refuge in app which doesn't depend on llama-cpp-python..

ollama uses llama.cpp, and running as expected.. not 10x slower than usual which if I can even get llama-cpp-python to work is 4 minute summary rather than 40 seconds

[tbatchelli]

For what it's worth, passing offload_kqv=True to llama-cpp-python fixed it for me. I am using LangChain, so I am basically passing the following parameter to nangchain's LlamaCpp:

model_kwargs={"threads":1,
              "offload_kqv": True,}