Failing to get a response!

[arbianqx]

LocalAI version:
v1.40.0-cublas-cuda11

Environment, CPU architecture, OS, and Version:
Linux pop-os 6.0.12-76060012-generic #202212290932167406645920.04~3cd2bf3-Ubuntu SMP PREEMPT_DYNAMI x86_64 x86_64 x86_64 GNU/Linux

Describe the bug
I have installed successfully a model, by running:
curl http://localhost:8080/models/apply -H "Content-Type: application/json" -d '{
"id": "huggingface@TheBloke/Yarn-Mistral-7B-128k-GGUF/yarn-mistral-7b-128k.Q5_K_M.gguf"
}'

After that it just keeps hanging and no response was comming:
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "thebloke__yarn-mistral-7b-128k-gguf__yarn-mistral-7b-128k.q5_k_m.gguf",
"messages": [{"role": "user", "content": "Say this is a test!"}],
"temperature": 0.1
}'

To Reproduce
Run docker container as follows:
docker run -d --rm
--name api
--gpus '"device=1"'
-p 8080:8080
--env-file .env
-v $(pwd)/models:/models
-v $(pwd)/images:/tmp/generated/images
quay.io/go-skynet/local-ai:v1.40.0-cublas-cuda11
/usr/bin/local-ai

Expected behavior
I'm expecting to get a response, but nothing is happening.

Logs
logs.txt

Additional context
Here is my .env:
GALLERIES=[{"name":"model-gallery", "url":"github:go-skynet/model-gallery/index.yaml"}, {"url": "github:go-skynet/model-gallery/huggingface.yaml","name":"huggingface"}]

MODELS_PATH=/models

DEBUG=true

BUILD_TYPE=cublas

I have also tried running without GPU, but the same keeps happening. Any idea on this?

[localai-bot]

⚠️⚠️⚠️⚠️⚠️

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

but.... I can also be funny or helpful 😸 and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me,
I don't support (yet) replying!

⚠️⚠️⚠️⚠️⚠️

It seems like there might be a problem with the model or the way it's being loaded. Can you try loading a different model and see if that works? If not, can you provide the logs from when you run curl http://localhost:8080/models/apply -H "Content-Type: application/json" -d '{"id": "huggingface@TheBloke/Yarn-Mistral-7B-128k-GGUF/yarn-mistral-7b-128k.Q5_K_M.gguf"}'?

Sources:

• https://localai.io/basics/getting_started/index.html
• https://github.com/go-skynet/LocalAI/tree/master/.github/ISSUE_TEMPLATE/bug_report.md
• https://github.com/go-skynet/LocalAI/tree/master/README.md

[Aisuko]

Hi @arbianqx, according to the log. I believe the CPU acceleration was failed, but I am not 100% confirm it.

11:16AM DBG GRPC(yarn-mistral-7b-128k.Q5_K_M.gguf-127.0.0.1:39895): stderr llm_load_tensors: using CUDA for GPU acceleration
11:16AM DBG GRPC(yarn-mistral-7b-128k.Q5_K_M.gguf-127.0.0.1:39895): stderr llm_load_tensors: mem required  = 4893.09 MB
11:16AM DBG GRPC(yarn-mistral-7b-128k.Q5_K_M.gguf-127.0.0.1:39895): stderr llm_load_tensors: offloading 0 repeating layers to GPU
11:16AM DBG GRPC(yarn-mistral-7b-128k.Q5_K_M.gguf-127.0.0.1:39895): stderr llm_load_tensors: offloaded 0/35 layers to GPU
11:16AM DBG GRPC(yarn-mistral-7b-128k.Q5_K_M.gguf-127.0.0.1:39895): stderr llm_load_tensors: VRAM used: 0.00 MB

Have you check the GPU usage during the inference ? How much memory total of the GPU?

I assume only CPU with no any other features open. The inference process would be so slow to response.

[lunamidori5]

@arbianqx if you would like GPU support you will need to add that to the models yaml file see here on how to do that - https://localai.io/howtos/easy-model-import-downloaded/

[arbianqx]

@lunamidori5 I have already completed those examples. I went by the docs step by step. It seems there's something missing

[lunamidori5]

@arbianqx models yaml file please <3

[arbianqx]

@lunamidori5 Yes sure. My plan is to use Mistral Models or such, but I started with the sample provided, by docs.

backend: llama context_size: 2000 name: lunademo parameters: model: luna-ai-llama2-uncensored.Q4_K_M.gguf temperature: 0.2 top_k: 40 top_p: 0.65 roles: assistant: 'ASSISTANT:' system: 'SYSTEM:' user: 'USER:' template: chat: lunademo-chat completion: lunademo-completion f16: true gpu_layers: 4

[lunamidori5]

@arbianqx so both your host and docker have cuda installed? Can you please post your openai v0 request?

[arbianqx]

@lunamidori5 Yupp cuda and docker. I was using the curl option provided by docs.

[lunamidori5]

@arbianqx this one? there is like... 5 in the how tos, so just making sure

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
     "model": "lunademo",
     "messages": [{"role": "user", "content": "How are you?"}],
     "temperature": 0.9 
   }'

[arbianqx]

@lunamidori5 I'm sorry if I'm not providing all the infos. Yes I was using this sample.

[lunamidori5]

@arbianqx Your good!! Alright one moment let me look everything over