Need long time to generate text, any body can help?

[hanifabd]

I used google colab cpu to try it. this is my code. i am trying to use larger context but this take so long to generate the text

llm = Llama(model_path="/content/ggml-vicuna-13b-4bit-rev1.bin", n_ctx=2048, n_threads=20, n_batch=2048)

output = llm(
    "Question: Any suggestion for delicious food? Answer:",
    max_tokens=1024,
    temperature=0.9,
    top_p=0.95,
    repeat_penalty=1.2,
    top_k=50,
    echo=True
)

print(output)

[abetlen]

How many cpus are in the colab notebook? There's an issue in llama.cpp that causes extreme slowdown when n_threads > cpu_count

[hanifabd]

with this code i got 2 cpus, with your info, so i need to reduce my n_threads, isn'it?

[abetlen]

Correct, if the colab notebook supports openblas then you may be able to speed up eval if you install with LLAMA_OPENBLAS=1 pip install llama-cpp-python or if CUDA and cuBLAS are installed LLAMA_CUBLAS=1 pip install llama-cpp-python

[hanifabd]

I have try to reduce n_threads to 1 and 2 but still facing the same issue. then i try to install the openblas in my colab and try to reduce the n_batch to 1024 but still same. the generation took so long even more than 10 minutes

llm = Llama(model_path="/content/ggml-vicuna-13b-4bit-rev1.bin", n_ctx=2048, n_threads=1, n_batch=1024)

output = llm(
    "Question: Any suggestion for delicious food? Answer:",
    max_tokens=1024,
    temperature=0.9,
    top_p=0.95,
    repeat_penalty=1.2,
    top_k=50,
    echo=True
)

*edited

39 minutes for that code

[mzbac]

Somehow it is very slow comparing with the native llama.cpp. not very sure why.

[hanifabd]

Yuppp, i have tried for shorter context but quiet slow. context 32/64 will generate for about 2-5 minutes. let me inform here if i find something usefull. or maybe if someone know something let me know and share here please. thank you

[gjmulder]

Any update?

[gjmulder]

#232 looks to be the best issue for analysing performance