LocalAI returns Server error error="could not load model: rpc error: code = Unavailable desc = error reading from server: EOF"

[CyberGWJ]

LocalAI version:
Fresh intall latest version

Environment, CPU architecture, OS, and Version:
Proxmox VM: Linux localAI 6.1.0-22-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.94-1 (2024-06-21) x86_64 GNU/Linux

Describe the bug
When trying to chat using the statement below. Localai returns the error.

curl http://localhost:8989/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "Luna-AI-Llama2-Uncensored-GGUF",
"messages": [{"role": "user", "content": "How are you?"}],
"temperature": 0.9
}'

To Reproduce
Do a fresh install on a VM (Proxmox) and do the following steps.

1.  apt install curl git (git may not be needed but I normaly install it)
2.  curl https://localai.io/install.sh | PORT=8989 USE_AIO=false sh (I have tried USE_AIO=true and get the same results)
3. nano /etc/localai.env and add GALLERIES=[{"name":"model-gallery", "url":"github:go-skynet/model-gallery/index.yaml"}, {"url": "github:go-skynet/model-gallery/huggingface.yaml","name":"huggingface"}]
4. nano /usr/share/local-ai/models/Luna-AI-Llama2-Uncensored-GGUF.yaml

---

name: Luna-AI-Llama2-Uncensored-GGUF
context_size: 2048
trimsuffix:

• "\n"
  mmap: false
  parameters:
  model: huggingface://TheBloke/Luna-AI-Llama2-Uncensored-GGUF/luna-ai-llama2-uncensored.Q5_K_M.gguf
  top_k: 80
  temperature: 0.2
  top_p: 0.7
  backend: llama
  roles:
  assistant: 'ASSISTANT:'
  system: 'SYSTEM:'
  user: 'USER:'
  template:
  chat: lunademo-chat
  completion: lunademo-completion

---

5. nano /usr/share/local-ai/models/lunademo-chat.tmpl

---

USER: {{.Input}}

ASSISTANT:

6. nano /usr/share/local-ai/models/lunademo-completion.tmpl

---

Complete the following sentence: {{.Input}}

7. systemctl stop local-ai
8. local-ai --debug

Expected behavior
Should return a response to the chat

Logs
root@localAI:~# local-ai --debug
8:30AM INF env file found, loading environment variables from file envFile=/etc/localai.env
8:30AM DBG Setting logging to debug
8:30AM INF Starting LocalAI using 8 threads, with models path: /usr/share/local-ai/models
8:30AM INF LocalAI version: ()
8:30AM DBG CPU capabilities: [aes apic clflush cmov constant_tsc cpuid cpuid_fault cx16 cx8 de fpu fxsr ht hypervisor lahf_lm lm mca mce mmx msr mtrr nopl nx pae pat pge pni popcnt pse pse36 pti sep sse sse2 sse4_1 sse4_2 ssse3 syscall tsc tsc_known_freq x2apic xtopology]
8:30AM DBG GPU count: 1
8:30AM DBG GPU: card #0 @0000:00:02.0 -> driver: 'bochs-drm' class: 'Display controller' vendor: 'unknown' product: 'unknown'
8:30AM DBG guessDefaultsFromFile: template already set name=Luna-AI-Llama2-Uncensored-GGUF
8:30AM INF Preloading models from /usr/share/local-ai/models

Model name: Luna-AI-Llama2-Uncensored-GGUF

8:30AM DBG Model: Luna-AI-Llama2-Uncensored-GGUF (config: {PredictionOptions:{Model:f83553a34a79b75aca661acbf73b8d62 Language: Translate:false N:0 TopP:0xc000cf8de0 TopK:0xc000cf8db8 Temperature:0xc000cf8dc0 Maxtokens:0xc000cf8e98 Echo:false Batch:0 IgnoreEOS:false RepeatPenalty:0 RepeatLastN:0 Keep:0 FrequencyPenalty:0 PresencePenalty:0 TFZ:0xc000cf8e90 TypicalP:0xc000cf8e88 Seed:0xc000cf8eb0 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:Luna-AI-Llama2-Uncensored-GGUF F16:0xc000cf8e50 Threads:0xc000cf8e48 Debug:0xc000cf8ea8 Roles:map[assistant:ASSISTANT: system:SYSTEM: user:USER:] Embeddings:false Backend:llama TemplateConfig:{Chat:lunademo-chat ChatMessage: Completion:lunademo-completion Edit: Functions: UseTokenizerTemplate:false JoinChatMessagesByCharacter:} PromptStrings:[] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: ResponseFormat: ResponseFormatMap:map[] FunctionsConfig:{DisableNoAction:false GrammarConfig:{ParallelCalls:false DisableParallelNewLines:false MixedMode:false NoMixedFreeString:false NoGrammar:false Prefix: ExpectStringsAfterJSON:false PropOrder:} NoActionFunctionName: NoActionDescriptionName: ResponseRegex:[] JSONRegexMatch:[] ReplaceFunctionResults:[] ReplaceLLMResult:[] CaptureLLMResult:[] FunctionName:false} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0xc000cf8e80 MirostatTAU:0xc000cf8e78 Mirostat:0xc000cf8e70 NGPULayers:0xc000cf8ea0 MMap:0xc000cf8cb8 MMlock:0xc000cf8ea9 LowVRAM:0xc000cf8ea9 Grammar: StopWords:[] Cutstrings:[] TrimSpace:[] TrimSuffix:[
] ContextSize:0xc000cf8ca8 NUMA:false LoraAdapter: LoraBase: LoraScale:0 NoMulMatQ:false DraftModel: NDraft:0 Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: FlashAttention:false NoKVOffloading:false RopeScaling: ModelType: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{CUDA:false PipelineType: SchedulerType: EnableParameters: CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder: ControlNet:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} TTSConfig:{Voice: VallE:{AudioPath:}} CUDA:false DownloadFiles:[] Description: Usage:})
8:30AM DBG Extracting backend assets files to /tmp/localai/backend_data
8:30AM DBG processing api keys runtime update
8:30AM DBG processing external_backends.json
8:30AM DBG external backends loaded from external_backends.json
8:30AM INF core/startup process completed!
8:30AM DBG No configuration file found at /tmp/localai/upload/uploadedFiles.json
8:30AM DBG No configuration file found at /tmp/localai/config/assistants.json
8:30AM DBG No configuration file found at /tmp/localai/config/assistantsFile.json
8:30AM INF LocalAI API is listening! Please connect to the endpoint for API documentation. endpoint=http://0.0.0.0:8989
8:30AM DBG Request received: {"model":"Luna-AI-Llama2-Uncensored-GGUF","language":"","translate":false,"n":0,"top_p":null,"top_k":null,"temperature":0.9,"max_tokens":null,"echo":false,"batch":0,"ignore_eos":false,"repeat_penalty":0,"repeat_last_n":0,"n_keep":0,"frequency_penalty":0,"presence_penalty":0,"tfz":null,"typical_p":null,"seed":null,"negative_prompt":"","rope_freq_base":0,"rope_freq_scale":0,"negative_prompt_scale":0,"use_fast_tokenizer":false,"clip_skip":0,"tokenizer":"","file":"","size":"","prompt":null,"instruction":"","input":null,"stop":null,"messages":[{"role":"user","content":"How are you?"}],"functions":null,"function_call":null,"stream":false,"mode":0,"step":0,"grammar":"","grammar_json_functions":null,"grammar_json_name":null,"backend":"","model_base_name":""}
8:30AM DBG guessDefaultsFromFile: template already set name=Luna-AI-Llama2-Uncensored-GGUF
8:30AM DBG Configuration read: &{PredictionOptions:{Model:f83553a34a79b75aca661acbf73b8d62 Language: Translate:false N:0 TopP:0xc000cf8de0 TopK:0xc000cf8db8 Temperature:0xc000515930 Maxtokens:0xc000cf8e98 Echo:false Batch:0 IgnoreEOS:false RepeatPenalty:0 RepeatLastN:0 Keep:0 FrequencyPenalty:0 PresencePenalty:0 TFZ:0xc000cf8e90 TypicalP:0xc000cf8e88 Seed:0xc000cf8eb0 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:Luna-AI-Llama2-Uncensored-GGUF F16:0xc000cf8e50 Threads:0xc000cf8e48 Debug:0xc000515a20 Roles:map[assistant:ASSISTANT: system:SYSTEM: user:USER:] Embeddings:false Backend:llama TemplateConfig:{Chat:lunademo-chat ChatMessage: Completion:lunademo-completion Edit: Functions: UseTokenizerTemplate:false JoinChatMessagesByCharacter:} PromptStrings:[] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: ResponseFormat: ResponseFormatMap:map[] FunctionsConfig:{DisableNoAction:false GrammarConfig:{ParallelCalls:false DisableParallelNewLines:false MixedMode:false NoMixedFreeString:false NoGrammar:false Prefix: ExpectStringsAfterJSON:false PropOrder:} NoActionFunctionName: NoActionDescriptionName: ResponseRegex:[] JSONRegexMatch:[] ReplaceFunctionResults:[] ReplaceLLMResult:[] CaptureLLMResult:[] FunctionName:false} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0xc000cf8e80 MirostatTAU:0xc000cf8e78 Mirostat:0xc000cf8e70 NGPULayers:0xc000cf8ea0 MMap:0xc000cf8cb8 MMlock:0xc000cf8ea9 LowVRAM:0xc000cf8ea9 Grammar: StopWords:[] Cutstrings:[] TrimSpace:[] TrimSuffix:[
] ContextSize:0xc000cf8ca8 NUMA:false LoraAdapter: LoraBase: LoraScale:0 NoMulMatQ:false DraftModel: NDraft:0 Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: FlashAttention:false NoKVOffloading:false RopeScaling: ModelType: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{CUDA:false PipelineType: SchedulerType: EnableParameters: CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder: ControlNet:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} TTSConfig:{Voice: VallE:{AudioPath:}} CUDA:false DownloadFiles:[] Description: Usage:}
8:30AM DBG Parameters: &{PredictionOptions:{Model:f83553a34a79b75aca661acbf73b8d62 Language: Translate:false N:0 TopP:0xc000cf8de0 TopK:0xc000cf8db8 Temperature:0xc000515930 Maxtokens:0xc000cf8e98 Echo:false Batch:0 IgnoreEOS:false RepeatPenalty:0 RepeatLastN:0 Keep:0 FrequencyPenalty:0 PresencePenalty:0 TFZ:0xc000cf8e90 TypicalP:0xc000cf8e88 Seed:0xc000cf8eb0 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:Luna-AI-Llama2-Uncensored-GGUF F16:0xc000cf8e50 Threads:0xc000cf8e48 Debug:0xc000515a20 Roles:map[assistant:ASSISTANT: system:SYSTEM: user:USER:] Embeddings:false Backend:llama TemplateConfig:{Chat:lunademo-chat ChatMessage: Completion:lunademo-completion Edit: Functions: UseTokenizerTemplate:false JoinChatMessagesByCharacter:} PromptStrings:[] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: ResponseFormat: ResponseFormatMap:map[] FunctionsConfig:{DisableNoAction:false GrammarConfig:{ParallelCalls:false DisableParallelNewLines:false MixedMode:false NoMixedFreeString:false NoGrammar:false Prefix: ExpectStringsAfterJSON:false PropOrder:} NoActionFunctionName: NoActionDescriptionName: ResponseRegex:[] JSONRegexMatch:[] ReplaceFunctionResults:[] ReplaceLLMResult:[] CaptureLLMResult:[] FunctionName:false} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0xc000cf8e80 MirostatTAU:0xc000cf8e78 Mirostat:0xc000cf8e70 NGPULayers:0xc000cf8ea0 MMap:0xc000cf8cb8 MMlock:0xc000cf8ea9 LowVRAM:0xc000cf8ea9 Grammar: StopWords:[] Cutstrings:[] TrimSpace:[] TrimSuffix:[
] ContextSize:0xc000cf8ca8 NUMA:false LoraAdapter: LoraBase: LoraScale:0 NoMulMatQ:false DraftModel: NDraft:0 Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: FlashAttention:false NoKVOffloading:false RopeScaling: ModelType: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{CUDA:false PipelineType: SchedulerType: EnableParameters: CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder: ControlNet:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} TTSConfig:{Voice: VallE:{AudioPath:}} CUDA:false DownloadFiles:[] Description: Usage:}
8:30AM DBG Prompt (before templating): USER:How are you?
8:30AM DBG Template found, input modified to: USER: USER:How are you?

ASSISTANT:

8:30AM DBG Prompt (after templating): USER: USER:How are you?

ASSISTANT:

8:30AM INF Loading model 'f83553a34a79b75aca661acbf73b8d62' with backend llama
8:30AM DBG llama-cpp is an alias of llama-cpp
8:30AM DBG Loading model in memory from file: /usr/share/local-ai/models/f83553a34a79b75aca661acbf73b8d62
8:30AM DBG Loading Model f83553a34a79b75aca661acbf73b8d62 with gRPC (file: /usr/share/local-ai/models/f83553a34a79b75aca661acbf73b8d62) (backend: llama-cpp): {backendString:llama model:f83553a34a79b75aca661acbf73b8d62 threads:8 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc0009b3688 externalBackends:map[] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false parallelRequests:false}
8:30AM INF [llama-cpp] attempting to load with fallback variant
8:30AM DBG ld.so found
8:30AM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/lib/ld.so
8:30AM DBG GRPC Service for f83553a34a79b75aca661acbf73b8d62 will be running at: '127.0.0.1:44937'
8:30AM DBG GRPC Service state dir: /tmp/go-processmanager644594595
8:30AM DBG GRPC Service Started
8:30AM DBG GRPC(f83553a34a79b75aca661acbf73b8d62-127.0.0.1:44937): stdout Server listening on 127.0.0.1:44937
8:30AM DBG GRPC Service Ready
8:30AM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:} sizeCache:0 unknownFields:[] Model:f83553a34a79b75aca661acbf73b8d62 ContextSize:2048 Seed:1541718028 NBatch:512 F16Memory:false MLock:false MMap:false VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:99999999 MainGPU: TensorSplit: Threads:8 LibrarySearchPath: RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/usr/share/local-ai/models/f83553a34a79b75aca661acbf73b8d62 Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 ControlNet: Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0 Type: FlashAttention:false NoKVOffload:false}
8:30AM ERR Server error error="could not load model: rpc error: code = Unavailable desc = error reading from server: EOF" ip=127.0.0.1 latency=2.045800508s method=POST status=500 url=/v1/chat/completions

FYI This is my first time using localai so if I missed something please let me know. Thanks.

[CyberGWJ]

Here are the contents for the models directory

root@localAI:/usr/share/local-ai/models# ls -ial
total 4671076
3438895 drwxr-x--- 2 local-ai local-ai 4096 Jul 1 08:27 .
3438891 drwxr-xr-x 4 local-ai local-ai 4096 Jul 1 08:21 ..
3438897 -rw-r--r-- 1 root root 4783156960 Jul 1 08:27 f83553a34a79b75aca661acbf73b8d62
3438898 -rw-r--r-- 1 root root 394 Jul 1 08:24 Luna-AI-Llama2-Uncensored-GGUF.yaml
3438899 -rw-r--r-- 1 root root 30 Jul 1 08:24 lunademo-chat.tmpl
3438900 -rw-r--r-- 1 root root 44 Jul 1 08:25 lunademo-completion.tmpl

[mudler]

@CyberGWJ - that's quite an old model and setup I see in there. Can you just try with defaults, navigate to the WebUI and install models from there?

[CyberGWJ]

Yes, I will start over and post the results. Thanks

[CyberGWJ]

@mudler I am trying to do a fresh install and am getting error no matter what options I enter.

root@localAI:~# curl https://localai.io/install.sh | PORT=8989 USE_AIO=true DOCKER_INSTALL=false sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 20078 0 20078 0 0 52851 0 --:--:-- --:--:-- --:--:-- 52976
sh: 585: [: Linux: unexpected operator
Downloading local-ai...
curl: (22) The requested URL returned error: 404

root@localAI:~# curl https://localai.io/install.sh | sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 20078 0 20078 0 0 111k 0 --:--:-- --:--:-- --:--:-- 112k
sh: 585: [: Linux: unexpected operator
Downloading local-ai...
curl: (22) The requested URL returned error: 404

root@localAI:~# curl https://localai.io/install.sh | PORT=8989 USE_AIO=true DOCKER_INSTALL=false sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 20078 0 20078 0 0 97586 0 --:--:-- --:--:-- --:--:-- 97941
sh: 585: [: Linux: unexpected operator
Downloading local-ai...
curl: (22) The requested URL returned error: 404

root@localAI:~#

[CyberGWJ]

@mudler I see the problem. There is a new release yet the local-ai-Linux-x86_64 version is not there yet.

[CyberGWJ]

@mudler Here is an update. I saw the new release was posted and went through the following steps.

LocalAI version:
Fresh intall os VM and localai latest version

Environment, CPU architecture, OS, and Version:
Proxmox VM: Linux localAI 6.1.0-22-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.94-1 (2024-06-21) x86_64 GNU/Linux

Describe the bug
When trying to chat using the statement below. Localai returns the error.

curl http://localhost:8989/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "phi-2-chat:Q8_0",
"messages": [{"role": "user", "content": "How are you doing?", "temperature": 0.1}]
}'

{"error":{"code":500,"message":"could not load model - all backends returned error: [llama-cpp]: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF\n[llama-cpp]: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF\n[llama-ggml]: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF\n[gpt4all]: could not load model: rpc error: code = Unknown desc = failed loading model\n[llama-cpp-fallback]: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF\n[rwkv]: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF\n[whisper]: grpc service not ready\n[huggingface]: could not load model: rpc error: code = Unknown desc = no huggingface token provided\n[bert-embeddings]: could not load model: rpc error: code = Unknown desc = failed loading model","type":""}}

To Reproduce
Do a fresh install on a VM (Proxmox) and do the following steps.

1. apt install curl git (git may not be needed but I normaly install it)
2. curl https://localai.io/install.sh | sh
3. http://ipaddress:8080
4. Go to Models and install phi-2-chat:Q8_0
5. systemctl stop local-ai
6. nano /usr/share/local-ai/models/phi-2-chat:Q8_0.yaml
   Note: I replace the yaml statements with the one on the localai github and use the name from the original yaml file
7. local-ai --debug

Output from debug session.

root@localAI:~# local-ai --debug
9:38PM INF env file found, loading environment variables from file envFile=/etc/localai.env
9:38PM DBG Setting logging to debug
9:38PM INF Starting LocalAI using 8 threads, with models path: /usr/share/local-ai/models
9:38PM INF LocalAI version: ()
9:38PM DBG CPU capabilities: [aes apic clflush cmov constant_tsc cpuid cpuid_fault cx16 cx8 de fpu fxsr ht hypervisor lahf_lm lm mca mce mmx msr mtrr nopl nx pae pat pge pni popcnt pse pse36 pti sep sse sse2 sse4_1 sse4_2 ssse3 syscall tsc tsc_known_freq x2apic xtopology]
9:38PM DBG GPU count: 1
9:38PM DBG GPU: card #0 @0000:00:02.0 -> driver: 'bochs-drm' class: 'Display controller' vendor: 'unknown' product: 'unknown'
9:38PM DBG guessDefaultsFromFile: template already set name=phi-2-chat:Q8_0
9:38PM INF Preloading models from /usr/share/local-ai/models

Model name: phi-2-chat:Q8_0

curl http://localhost:8080/v1/chat/completions -H "Content-Type:
application/json" -d '{ "model": "phi-2-chat", "messages": [{"role": "user",
"content": "How are you doing?", "temperature": 0.1}] }'

9:38PM DBG Model: phi-2-chat:Q8_0 (config: {PredictionOptions:{Model:phi-2-layla-v1-chatml-Q8_0.gguf Language: Translate:false N:0 TopP:0xc000a4ad88 TopK:0xc000a4ad90 Temperature:0xc000a4ad98 Maxtokens:0xc000a4adc8 Echo:false Batch:0 IgnoreEOS:false RepeatPenalty:0 RepeatLastN:0 Keep:0 FrequencyPenalty:0 PresencePenalty:0 TFZ:0xc000a4adc0 TypicalP:0xc000a4adb8 Seed:0xc000a4ade0 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:phi-2-chat:Q8_0 F16:0xc000a4ac30 Threads:0xc000a4ad78 Debug:0xc000a4add8 Roles:map[] Embeddings:false Backend: TemplateConfig:{Chat:{{.Input}}
<|im_start|>assistant
ChatMessage:<|im_start|>{{if eq .RoleName "assistant"}}assistant{{else if eq .RoleName "system"}}system{{else if eq .RoleName "user"}}user{{end}}
{{if .Content}}{{.Content}}{{end}}
<|im_end|>
Completion:{{.Input}}
Edit: Functions: UseTokenizerTemplate:false JoinChatMessagesByCharacter:} PromptStrings:[] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: ResponseFormat: ResponseFormatMap:map[] FunctionsConfig:{DisableNoAction:false GrammarConfig:{ParallelCalls:false DisableParallelNewLines:false MixedMode:false NoMixedFreeString:false NoGrammar:false Prefix: ExpectStringsAfterJSON:false PropOrder:} NoActionFunctionName: NoActionDescriptionName: ResponseRegex:[] JSONRegexMatch:[] ReplaceFunctionResults:[] ReplaceLLMResult:[] CaptureLLMResult:[] FunctionName:false} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0xc000a4adb0 MirostatTAU:0xc000a4ada8 Mirostat:0xc000a4ada0 NGPULayers:0xc000a4add0 MMap:0xc000a4ac18 MMlock:0xc000a4add9 LowVRAM:0xc000a4add9 Grammar: StopWords:[<|im_end|> ] Cutstrings:[] TrimSpace:[] TrimSuffix:[] ContextSize:0xc000a4ac20 NUMA:false LoraAdapter: LoraBase: LoraScale:0 NoMulMatQ:false DraftModel: NDraft:0 Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: FlashAttention:false NoKVOffloading:false RopeScaling: ModelType: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{CUDA:false PipelineType: SchedulerType: EnableParameters: CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder: ControlNet:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} TTSConfig:{Voice: VallE:{AudioPath:}} CUDA:false DownloadFiles:[] Description: Usage:curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "phi-2-chat",
"messages": [{"role": "user", "content": "How are you doing?", "temperature": 0.1}]
}'
})
9:38PM DBG Extracting backend assets files to /tmp/localai/backend_data
9:38PM DBG processing api keys runtime update
9:38PM DBG processing external_backends.json
9:38PM DBG external backends loaded from external_backends.json
9:38PM INF core/startup process completed!
9:38PM DBG No configuration file found at /tmp/localai/upload/uploadedFiles.json
9:38PM DBG No configuration file found at /tmp/localai/config/assistants.json
9:38PM DBG No configuration file found at /tmp/localai/config/assistantsFile.json
9:38PM INF LocalAI API is listening! Please connect to the endpoint for API documentation. endpoint=http://0.0.0.0:8080
9:38PM DBG Request received: {"model":"phi-2-chat:Q8_0","language":"","translate":false,"n":0,"top_p":null,"top_k":null,"temperature":null,"max_tokens":null,"echo":false,"batch":0,"ignore_eos":false,"repeat_penalty":0,"repeat_last_n":0,"n_keep":0,"frequency_penalty":0,"presence_penalty":0,"tfz":null,"typical_p":null,"seed":null,"negative_prompt":"","rope_freq_base":0,"rope_freq_scale":0,"negative_prompt_scale":0,"use_fast_tokenizer":false,"clip_skip":0,"tokenizer":"","file":"","size":"","prompt":null,"instruction":"","input":null,"stop":null,"messages":[{"role":"user","content":"How are you doing?"}],"functions":null,"function_call":null,"stream":false,"mode":0,"step":0,"grammar":"","grammar_json_functions":null,"grammar_json_name":null,"backend":"","model_base_name":""}
9:38PM DBG guessDefaultsFromFile: template already set name=phi-2-chat:Q8_0
9:38PM DBG Configuration read: &{PredictionOptions:{Model:phi-2-layla-v1-chatml-Q8_0.gguf Language: Translate:false N:0 TopP:0xc000a4ad88 TopK:0xc000a4ad90 Temperature:0xc000a4ad98 Maxtokens:0xc000a4adc8 Echo:false Batch:0 IgnoreEOS:false RepeatPenalty:0 RepeatLastN:0 Keep:0 FrequencyPenalty:0 PresencePenalty:0 TFZ:0xc000a4adc0 TypicalP:0xc000a4adb8 Seed:0xc000a4ade0 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:phi-2-chat:Q8_0 F16:0xc000a4ac30 Threads:0xc000a4ad78 Debug:0xc000529a60 Roles:map[] Embeddings:false Backend: TemplateConfig:{Chat:{{.Input}}
<|im_start|>assistant
ChatMessage:<|im_start|>{{if eq .RoleName "assistant"}}assistant{{else if eq .RoleName "system"}}system{{else if eq .RoleName "user"}}user{{end}}
{{if .Content}}{{.Content}}{{end}}
<|im_end|>
Completion:{{.Input}}
Edit: Functions: UseTokenizerTemplate:false JoinChatMessagesByCharacter:} PromptStrings:[] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: ResponseFormat: ResponseFormatMap:map[] FunctionsConfig:{DisableNoAction:false GrammarConfig:{ParallelCalls:false DisableParallelNewLines:false MixedMode:false NoMixedFreeString:false NoGrammar:false Prefix: ExpectStringsAfterJSON:false PropOrder:} NoActionFunctionName: NoActionDescriptionName: ResponseRegex:[] JSONRegexMatch:[] ReplaceFunctionResults:[] ReplaceLLMResult:[] CaptureLLMResult:[] FunctionName:false} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0xc000a4adb0 MirostatTAU:0xc000a4ada8 Mirostat:0xc000a4ada0 NGPULayers:0xc000a4add0 MMap:0xc000a4ac18 MMlock:0xc000a4add9 LowVRAM:0xc000a4add9 Grammar: StopWords:[<|im_end|> ] Cutstrings:[] TrimSpace:[] TrimSuffix:[] ContextSize:0xc000a4ac20 NUMA:false LoraAdapter: LoraBase: LoraScale:0 NoMulMatQ:false DraftModel: NDraft:0 Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: FlashAttention:false NoKVOffloading:false RopeScaling: ModelType: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{CUDA:false PipelineType: SchedulerType: EnableParameters: CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder: ControlNet:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} TTSConfig:{Voice: VallE:{AudioPath:}} CUDA:false DownloadFiles:[] Description: Usage:curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "phi-2-chat",
"messages": [{"role": "user", "content": "How are you doing?", "temperature": 0.1}]
}'
}
9:38PM DBG Parameters: &{PredictionOptions:{Model:phi-2-layla-v1-chatml-Q8_0.gguf Language: Translate:false N:0 TopP:0xc000a4ad88 TopK:0xc000a4ad90 Temperature:0xc000a4ad98 Maxtokens:0xc000a4adc8 Echo:false Batch:0 IgnoreEOS:false RepeatPenalty:0 RepeatLastN:0 Keep:0 FrequencyPenalty:0 PresencePenalty:0 TFZ:0xc000a4adc0 TypicalP:0xc000a4adb8 Seed:0xc000a4ade0 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:phi-2-chat:Q8_0 F16:0xc000a4ac30 Threads:0xc000a4ad78 Debug:0xc000529a60 Roles:map[] Embeddings:false Backend: TemplateConfig:{Chat:{{.Input}}
<|im_start|>assistant
ChatMessage:<|im_start|>{{if eq .RoleName "assistant"}}assistant{{else if eq .RoleName "system"}}system{{else if eq .RoleName "user"}}user{{end}}
{{if .Content}}{{.Content}}{{end}}
<|im_end|>
Completion:{{.Input}}
Edit: Functions: UseTokenizerTemplate:false JoinChatMessagesByCharacter:} PromptStrings:[] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: ResponseFormat: ResponseFormatMap:map[] FunctionsConfig:{DisableNoAction:false GrammarConfig:{ParallelCalls:false DisableParallelNewLines:false MixedMode:false NoMixedFreeString:false NoGrammar:false Prefix: ExpectStringsAfterJSON:false PropOrder:} NoActionFunctionName: NoActionDescriptionName: ResponseRegex:[] JSONRegexMatch:[] ReplaceFunctionResults:[] ReplaceLLMResult:[] CaptureLLMResult:[] FunctionName:false} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0xc000a4adb0 MirostatTAU:0xc000a4ada8 Mirostat:0xc000a4ada0 NGPULayers:0xc000a4add0 MMap:0xc000a4ac18 MMlock:0xc000a4add9 LowVRAM:0xc000a4add9 Grammar: StopWords:[<|im_end|> ] Cutstrings:[] TrimSpace:[] TrimSuffix:[] ContextSize:0xc000a4ac20 NUMA:false LoraAdapter: LoraBase: LoraScale:0 NoMulMatQ:false DraftModel: NDraft:0 Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: FlashAttention:false NoKVOffloading:false RopeScaling: ModelType: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{CUDA:false PipelineType: SchedulerType: EnableParameters: CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder: ControlNet:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} TTSConfig:{Voice: VallE:{AudioPath:}} CUDA:false DownloadFiles:[] Description: Usage:curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "phi-2-chat",
"messages": [{"role": "user", "content": "How are you doing?", "temperature": 0.1}]
}'
}
9:38PM DBG templated message for chat: <|im_start|>user
How are you doing?
<|im_end|>

9:38PM DBG Prompt (before templating): <|im_start|>user
How are you doing?
<|im_end|>

9:38PM DBG Template found, input modified to: <|im_start|>user
How are you doing?
<|im_end|>

<|im_start|>assistant

9:38PM DBG Prompt (after templating): <|im_start|>user
How are you doing?
<|im_end|>

<|im_start|>assistant

9:38PM DBG Loading from the following backends (in order): [llama-cpp llama-ggml gpt4all llama-cpp-fallback rwkv whisper huggingface bert-embeddings]
9:38PM INF Trying to load the model 'phi-2-layla-v1-chatml-Q8_0.gguf' with the backend '[llama-cpp llama-ggml gpt4all llama-cpp-fallback rwkv whisper huggingface bert-embeddings]'
9:38PM INF [llama-cpp] Attempting to load
9:38PM INF Loading model 'phi-2-layla-v1-chatml-Q8_0.gguf' with backend llama-cpp
9:38PM DBG Loading model in memory from file: /usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf
9:38PM DBG Loading Model phi-2-layla-v1-chatml-Q8_0.gguf with gRPC (file: /usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf) (backend: llama-cpp): {backendString:llama-cpp model:phi-2-layla-v1-chatml-Q8_0.gguf threads:8 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc000af6908 externalBackends:map[] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false parallelRequests:false}
9:38PM INF [llama-cpp] attempting to load with fallback variant
9:38PM DBG ld.so found
9:38PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/lib/ld.so
9:38PM DBG GRPC Service for phi-2-layla-v1-chatml-Q8_0.gguf will be running at: '127.0.0.1:41001'
9:38PM DBG GRPC Service state dir: /tmp/go-processmanager163597972
9:38PM DBG GRPC Service Started
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:41001): stdout Server listening on 127.0.0.1:41001
9:38PM DBG GRPC Service Ready
9:38PM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:} sizeCache:0 unknownFields:[] Model:phi-2-layla-v1-chatml-Q8_0.gguf ContextSize:4096 Seed:2052651737 NBatch:512 F16Memory:true MLock:false MMap:true VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:99999999 MainGPU: TensorSplit: Threads:8 LibrarySearchPath: RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 ControlNet: Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0 Type: FlashAttention:false NoKVOffload:false}
9:38PM INF [llama-cpp] Fails: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF
9:38PM INF [llama-cpp] Autodetection failed, trying the fallback
9:38PM INF Loading model 'phi-2-layla-v1-chatml-Q8_0.gguf' with backend llama-cpp-fallback
9:38PM DBG Loading model in memory from file: /usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf
9:38PM DBG Loading Model phi-2-layla-v1-chatml-Q8_0.gguf with gRPC (file: /usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf) (backend: llama-cpp-fallback): {backendString:llama-cpp-fallback model:phi-2-layla-v1-chatml-Q8_0.gguf threads:8 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc000af6908 externalBackends:map[] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false parallelRequests:false}
9:38PM DBG ld.so found
9:38PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/lib/ld.so
9:38PM DBG GRPC Service for phi-2-layla-v1-chatml-Q8_0.gguf will be running at: '127.0.0.1:43575'
9:38PM DBG GRPC Service state dir: /tmp/go-processmanager789209016
9:38PM DBG GRPC Service Started
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:43575): stdout Server listening on 127.0.0.1:43575
9:38PM DBG GRPC Service Ready
9:38PM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:} sizeCache:0 unknownFields:[] Model:phi-2-layla-v1-chatml-Q8_0.gguf ContextSize:4096 Seed:2052651737 NBatch:512 F16Memory:true MLock:false MMap:true VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:99999999 MainGPU: TensorSplit: Threads:8 LibrarySearchPath: RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 ControlNet: Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0 Type: FlashAttention:false NoKVOffload:false}
9:38PM INF [llama-cpp] Fails: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF
9:38PM INF [llama-ggml] Attempting to load
9:38PM INF Loading model 'phi-2-layla-v1-chatml-Q8_0.gguf' with backend llama-ggml
9:38PM DBG Loading model in memory from file: /usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf
9:38PM DBG Loading Model phi-2-layla-v1-chatml-Q8_0.gguf with gRPC (file: /usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf) (backend: llama-ggml): {backendString:llama-ggml model:phi-2-layla-v1-chatml-Q8_0.gguf threads:8 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc000af6908 externalBackends:map[] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false parallelRequests:false}
9:38PM DBG ld.so found
9:38PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/lib/ld.so
9:38PM DBG GRPC Service for phi-2-layla-v1-chatml-Q8_0.gguf will be running at: '127.0.0.1:39371'
9:38PM DBG GRPC Service state dir: /tmp/go-processmanager337258585
9:38PM DBG GRPC Service Started
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr 2024/07/01 21:38:33 gRPC Server listening at 127.0.0.1:39371
9:38PM DBG GRPC Service Ready
9:38PM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:} sizeCache:0 unknownFields:[] Model:phi-2-layla-v1-chatml-Q8_0.gguf ContextSize:4096 Seed:2052651737 NBatch:512 F16Memory:true MLock:false MMap:true VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:99999999 MainGPU: TensorSplit: Threads:8 LibrarySearchPath: RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 ControlNet: Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0 Type: FlashAttention:false NoKVOffload:false}
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr SIGILL: illegal instruction
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr PC=0x84e53a m=3 sigcode=2
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr signal arrived during cgo execution
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr instruction bytes: 0xc4 0xc1 0x79 0x7e 0xd7 0x41 0x56 0x41 0x55 0x41 0x54 0x49 0x89 0xfc 0x53 0xc5
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr goroutine 38 gp=0xc0000a6fc0 m=3 mp=0xc000063008 [syscall]:
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.cgocall(0x82b1e0, 0xc0002c75e8)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/cgocall.go:157 +0x4b fp=0xc0002c75c0 sp=0xc0002c7588 pc=0x4145eb
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr github.com/go-skynet/go-llama%2ecpp._Cfunc_load_model(0x7fc66c000cb0, 0x1000, 0x0, 0x1, 0x0, 0x0, 0x1, 0x0, 0x5f5e0ff, 0x200, ...)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr _cgo_gotypes.go:254 +0x4c fp=0xc0002c75e8 sp=0xc0002c75c0 pc=0x50d14c
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr github.com/go-skynet/go-llama%2ecpp.New({0xc0002b2040, 0x3a}, {0xc00028be80, 0x9, 0x9102c0?})
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/_work/LocalAI/LocalAI/sources/go-llama.cpp/llama.go:28 +0x28a fp=0xc0002c7768 sp=0xc0002c75e8 pc=0x50d8ea
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr main.(*LLM).Load(0xc000030e80, 0xc0002c0008)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/_work/LocalAI/LocalAI/backend/go/llm/llama-ggml/llama.go:73 +0x92e fp=0xc0002c78a8 sp=0xc0002c7768 pc=0x8286ae
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr github.com/mudler/LocalAI/pkg/grpc.(*server).LoadModel(0xc000030f40, {0x98f560?, 0xc00006d990?}, 0xc0002c0008)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/_work/LocalAI/LocalAI/pkg/grpc/server.go:50 +0xe7 fp=0xc0002c7950 sp=0xc0002c78a8 pc=0x825407
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr github.com/mudler/LocalAI/pkg/grpc/proto._Backend_LoadModel_Handler({0x98f560, 0xc000030f40}, {0xa6e638, 0xc0002865a0}, 0xc00028a080, 0x0)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/_work/LocalAI/LocalAI/pkg/grpc/proto/backend_grpc.pb.go:352 +0x1a6 fp=0xc0002c79a0 sp=0xc0002c7950 pc=0x81da06
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001b2200, {0xa6e638, 0xc000286510}, {0xa71f60, 0xc0002a6000}, 0xc0002ba000, 0xc0001c0de0, 0xd8ff70, 0x0)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/google.golang.org/grpc@v1.64.0/server.go:1379 +0xdf8 fp=0xc0002c7da0 sp=0xc0002c79a0 pc=0x804098
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr google.golang.org/grpc.(*Server).handleStream(0xc0001b2200, {0xa71f60, 0xc0002a6000}, 0xc0002ba000)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/google.golang.org/grpc@v1.64.0/server.go:1790 +0xe8b fp=0xc0002c7f78 sp=0xc0002c7da0 pc=0x808f6b
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr google.golang.org/grpc.(*Server).serveStreams.func2.1()
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/google.golang.org/grpc@v1.64.0/server.go:1029 +0x8b fp=0xc0002c7fe0 sp=0xc0002c7f78 pc=0x80214b
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.goexit({})
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0002c7fe8 sp=0xc0002c7fe0 pc=0x47c8c1
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 37
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/google.golang.org/grpc@v1.64.0/server.go:1040 +0x125
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr goroutine 1 gp=0xc000006380 m=nil [IO wait]:
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.gopark(0x1?, 0xc00019fb80?, 0xd1?, 0x6f?, 0xc00019fb38?)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/proc.go:402 +0xce fp=0xc00019fb00 sp=0xc00019fae0 pc=0x44ac0e
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.netpollblock(0x0?, 0x413d86?, 0x0?)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/netpoll.go:573 +0xf7 fp=0xc00019fb38 sp=0xc00019fb00 pc=0x443a17
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr internal/poll.runtime_pollWait(0x7fc6789c5760, 0x72)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/netpoll.go:345 +0x85 fp=0xc00019fb58 sp=0xc00019fb38 pc=0x477785
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr internal/poll.(*pollDesc).wait(0x3?, 0x432fa5?, 0x0)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc00019fb80 sp=0xc00019fb58 pc=0x4e20a7
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr internal/poll.(*pollDesc).waitRead(...)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/internal/poll/fd_poll_runtime.go:89
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr internal/poll.(*FD).Accept(0xc0000f8600)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/internal/poll/fd_unix.go:611 +0x2ac fp=0xc00019fc28 sp=0xc00019fb80 pc=0x4e744c
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr net.(*netFD).accept(0xc0000f8600)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/fd_unix.go:172 +0x29 fp=0xc00019fce0 sp=0xc00019fc28 pc=0x570889
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr net.(*TCPListener).accept(0xc00007c400)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/tcpsock_posix.go:159 +0x1e fp=0xc00019fd08 sp=0xc00019fce0 pc=0x587c3e
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr net.(*TCPListener).Accept(0xc00007c400)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/tcpsock.go:327 +0x30 fp=0xc00019fd38 sp=0xc00019fd08 pc=0x586e30
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr google.golang.org/grpc.(*Server).Serve(0xc0001b2200, {0xa6dd40, 0xc00007c400})
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/google.golang.org/grpc@v1.64.0/server.go:885 +0x49e fp=0xc00019fe98 sp=0xc00019fd38 pc=0x800bfe
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr github.com/mudler/LocalAI/pkg/grpc.StartServer({0x7ffdb96e6cea?, 0xc000024100?}, {0xa74158, 0xc000030e80})
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/_work/LocalAI/LocalAI/pkg/grpc/server.go:226 +0x170 fp=0xc00019ff20 sp=0xc00019fe98 pc=0x827ad0
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr main.main()
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/_work/LocalAI/LocalAI/backend/go/llm/llama-ggml/main.go:16 +0x85 fp=0xc00019ff50 sp=0xc00019ff20 pc=0x82a705
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.main()
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/proc.go:271 +0x29d fp=0xc00019ffe0 sp=0xc00019ff50 pc=0x44a7dd
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.goexit({})
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00019ffe8 sp=0xc00019ffe0 pc=0x47c8c1
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr goroutine 2 gp=0xc000006e00 m=nil [force gc (idle)]:
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/proc.go:402 +0xce fp=0xc00005cfa8 sp=0xc00005cf88 pc=0x44ac0e
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.goparkunlock(...)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/proc.go:408
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.forcegchelper()
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/proc.go:326 +0xb3 fp=0xc00005cfe0 sp=0xc00005cfa8 pc=0x44aa93
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.goexit({})
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00005cfe8 sp=0xc00005cfe0 pc=0x47c8c1
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr created by runtime.init.6 in goroutine 1
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/proc.go:314 +0x1a
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr goroutine 3 gp=0xc000007340 m=nil [GC sweep wait]:
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/proc.go:402 +0xce fp=0xc00005d780 sp=0xc00005d760 pc=0x44ac0e
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.goparkunlock(...)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/proc.go:408
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.bgsweep(0xc00007e000)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/mgcsweep.go:278 +0x94 fp=0xc00005d7c8 sp=0xc00005d780 pc=0x436214
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.gcenable.gowrap1()
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/mgc.go:203 +0x25 fp=0xc00005d7e0 sp=0xc00005d7c8 pc=0x42ab65
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.goexit({})
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00005d7e8 sp=0xc00005d7e0 pc=0x47c8c1
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr created by runtime.gcenable in goroutine 1
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/mgc.go:203 +0x66
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr goroutine 4 gp=0xc000007500 m=nil [GC scavenge wait]:
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.gopark(0xc00007e000?, 0xa667a0?, 0x1?, 0x0?, 0xc000007500?)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/proc.go:402 +0xce fp=0xc00005df78 sp=0xc00005df58 pc=0x44ac0e
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.goparkunlock(...)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/proc.go:408
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.(*scavengerState).park(0xdd9980)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00005dfa8 sp=0xc00005df78 pc=0x433c09
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.bgscavenge(0xc00007e000)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc00005dfc8 sp=0xc00005dfa8 pc=0x43419c
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.gcenable.gowrap2()
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/mgc.go:204 +0x25 fp=0xc00005dfe0 sp=0xc00005dfc8 pc=0x42ab05
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.goexit({})
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00005dfe8 sp=0xc00005dfe0 pc=0x47c8c1
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr created by runtime.gcenable in goroutine 1
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/mgc.go:204 +0xa5
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr goroutine 5 gp=0xc0000a6000 m=nil [finalizer wait]:
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.gopark(0xc00005c648?, 0x41e245?, 0xa8?, 0x1?, 0xc000006380?)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/proc.go:402 +0xce fp=0xc00005c620 sp=0xc00005c600 pc=0x44ac0e
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.runfinq()
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/mfinal.go:194 +0x107 fp=0xc00005c7e0 sp=0xc00005c620 pc=0x429ba7
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.goexit({})
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00005c7e8 sp=0xc00005c7e0 pc=0x47c8c1
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr created by runtime.createfing in goroutine 1
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/mfinal.go:164 +0x3d
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr goroutine 35 gp=0xc0002ae000 m=nil [select]:
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.gopark(0xc0001f9ea0?, 0x2?, 0x1e?, 0x0?, 0xc0001f9e74?)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/proc.go:402 +0xce fp=0xc0001f9d20 sp=0xc0001f9d00 pc=0x44ac0e
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.selectgo(0xc0001f9ea0, 0xc0001f9e70, 0x79b936?, 0x0, 0xc000298000?, 0x1)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/select.go:327 +0x725 fp=0xc0001f9e40 sp=0xc0001f9d20 pc=0x45bf85
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000296050, 0x1)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/google.golang.org/grpc@v1.64.0/internal/transport/controlbuf.go:418 +0x113 fp=0xc0001f9ed0 sp=0xc0001f9e40 pc=0x77ae53
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0001b0540)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/google.golang.org/grpc@v1.64.0/internal/transport/controlbuf.go:552 +0x86 fp=0xc0001f9f30 sp=0xc0001f9ed0 pc=0x77b5c6
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr google.golang.org/grpc/internal/transport.NewServerTransport.func2()
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/google.golang.org/grpc@v1.64.0/internal/transport/http2_server.go:334 +0xc7 fp=0xc0001f9fe0 sp=0xc0001f9f30 pc=0x791dc7
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.goexit({})
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0001f9fe8 sp=0xc0001f9fe0 pc=0x47c8c1
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 34
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/google.golang.org/grpc@v1.64.0/internal/transport/http2_server.go:332 +0x193e
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr goroutine 36 gp=0xc0002ae1c0 m=nil [select]:
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.gopark(0xc000058740?, 0x4?, 0x51?, 0xa6?, 0xc0000586c0?)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/proc.go:402 +0xce fp=0xc000058558 sp=0xc000058538 pc=0x44ac0e
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.selectgo(0xc000058740, 0xc0000586b8, 0x0?, 0x0, 0x0?, 0x1)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/select.go:327 +0x725 fp=0xc000058678 sp=0xc000058558 pc=0x45bf85
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0002a6000)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/google.golang.org/grpc@v1.64.0/internal/transport/http2_server.go:1168 +0x205 fp=0xc0000587c8 sp=0xc000058678 pc=0x798fc5
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr google.golang.org/grpc/internal/transport.NewServerTransport.gowrap1()
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/google.golang.org/grpc@v1.64.0/internal/transport/http2_server.go:355 +0x25 fp=0xc0000587e0 sp=0xc0000587c8 pc=0x791cc5
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.goexit({})
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000587e8 sp=0xc0000587e0 pc=0x47c8c1
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 34
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/google.golang.org/grpc@v1.64.0/internal/transport/http2_server.go:355 +0x1985
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr goroutine 37 gp=0xc0002ae380 m=nil [IO wait]:
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.gopark(0xc0002864b0?, 0xc0002ae380?, 0x18?, 0xd5?, 0xb?)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/proc.go:402 +0xce fp=0xc00006ca08 sp=0xc00006c9e8 pc=0x44ac0e
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.netpollblock(0x4c8118?, 0x413d86?, 0x0?)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/netpoll.go:573 +0xf7 fp=0xc00006ca40 sp=0xc00006ca08 pc=0x443a17
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr internal/poll.runtime_pollWait(0x7fc6789c5668, 0x72)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/netpoll.go:345 +0x85 fp=0xc00006ca60 sp=0xc00006ca40 pc=0x477785
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr internal/poll.(*pollDesc).wait(0xc00028a000?, 0xc00028e000?, 0x0)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc00006ca88 sp=0xc00006ca60 pc=0x4e20a7
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr internal/poll.(*pollDesc).waitRead(...)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/internal/poll/fd_poll_runtime.go:89
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr internal/poll.(*FD).Read(0xc00028a000, {0xc00028e000, 0x8000, 0x8000})
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/internal/poll/fd_unix.go:164 +0x27a fp=0xc00006cb20 sp=0xc00006ca88 pc=0x4e339a
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr net.(*netFD).Read(0xc00028a000, {0xc00028e000?, 0x1060100000000?, 0x8?})
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/fd_posix.go:55 +0x25 fp=0xc00006cb68 sp=0xc00006cb20 pc=0x56e8a5
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr net.(*conn).Read(0xc00028c000, {0xc00028e000?, 0x800010601?, 0xc000000000?})
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/net/net.go:179 +0x45 fp=0xc00006cbb0 sp=0xc00006cb68 pc=0x57ee45
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr net.(*TCPConn).Read(0x0?, {0xc00028e000?, 0xc00006cc08?, 0x46a5ed?})
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr :1 +0x25 fp=0xc00006cbe0 sp=0xc00006cbb0 pc=0x590f65
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr bufio.(*Reader).Read(0xc000288060, {0xc0002a0040, 0x9, 0xc000063808?})
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/bufio/bufio.go:241 +0x197 fp=0xc00006cc18 sp=0xc00006cbe0 pc=0x533f17
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr io.ReadAtLeast({0xa6b800, 0xc000288060}, {0xc0002a0040, 0x9, 0x9}, 0x9)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/io/io.go:335 +0x90 fp=0xc00006cc60 sp=0xc00006cc18 pc=0x4c1f50
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr io.ReadFull(...)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/io/io.go:354
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr golang.org/x/net/http2.readFrameHeader({0xc0002a0040, 0x9, 0xc000026588?}, {0xa6b800?, 0xc000288060?})
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/x/net@v0.25.0/http2/frame.go:237 +0x65 fp=0xc00006ccb0 sp=0xc00006cc60 pc=0x763c25
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr golang.org/x/net/http2.(*Framer).ReadFrame(0xc0002a0000)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/x/net@v0.25.0/http2/frame.go:501 +0x85 fp=0xc00006cd58 sp=0xc00006ccb0 pc=0x764365
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0002a6000, {0xa6e638, 0xc0002862a0}, 0xc0002862d0)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/google.golang.org/grpc@v1.64.0/internal/transport/http2_server.go:644 +0x15e fp=0xc00006ce88 sp=0xc00006cd58 pc=0x794cde
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr google.golang.org/grpc.(*Server).serveStreams(0xc0001b2200, {0xa6e5c8?, 0xeffce0?}, {0xa71f60, 0xc0002a6000}, {0xa716b8?, 0xc00028c000?})
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/google.golang.org/grpc@v1.64.0/server.go:1023 +0x3b6 fp=0xc00006cf70 sp=0xc00006ce88 pc=0x801ed6
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr google.golang.org/grpc.(*Server).handleRawConn.func1()
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/google.golang.org/grpc@v1.64.0/server.go:959 +0x56 fp=0xc00006cfe0 sp=0xc00006cf70 pc=0x801676
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr runtime.goexit({})
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.4.linux-amd64/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006cfe8 sp=0xc00006cfe0 pc=0x47c8c1
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 34
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr /home/runner/go/pkg/mod/google.golang.org/grpc@v1.64.0/server.go:958 +0x1c6
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr rax 0x1
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr rbx 0xc0002c75e8
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr rcx 0x1
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr rdx 0x0
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr rdi 0x7fc66c000cb0
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr rsi 0x1000
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr rbp 0x7fc67a1fdd50
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr rsp 0x7fc67a1fdd48
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr r8 0x0
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr r9 0x0
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr r10 0x0
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr r11 0x1
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr r12 0xc0002c8000
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr r13 0x200
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr r14 0xc0000a6fc0
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr r15 0x0
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr rip 0x84e53a
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr rflags 0x10216
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr cs 0x33
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr fs 0x0
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:39371): stderr gs 0x0
9:38PM INF [llama-ggml] Fails: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF
9:38PM INF [gpt4all] Attempting to load
9:38PM INF Loading model 'phi-2-layla-v1-chatml-Q8_0.gguf' with backend gpt4all
9:38PM DBG Loading model in memory from file: /usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf
9:38PM DBG Loading Model phi-2-layla-v1-chatml-Q8_0.gguf with gRPC (file: /usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf) (backend: gpt4all): {backendString:gpt4all model:phi-2-layla-v1-chatml-Q8_0.gguf threads:8 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc000af6908 externalBackends:map[] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false parallelRequests:false}
9:38PM DBG ld.so found
9:38PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/lib/ld.so
9:38PM DBG GRPC Service for phi-2-layla-v1-chatml-Q8_0.gguf will be running at: '127.0.0.1:46517'
9:38PM DBG GRPC Service state dir: /tmp/go-processmanager3806348243
9:38PM DBG GRPC Service Started
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:46517): stderr 2024/07/01 21:38:35 gRPC Server listening at 127.0.0.1:46517
9:38PM DBG GRPC Service Ready
9:38PM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:} sizeCache:0 unknownFields:[] Model:phi-2-layla-v1-chatml-Q8_0.gguf ContextSize:4096 Seed:2052651737 NBatch:512 F16Memory:true MLock:false MMap:true VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:99999999 MainGPU: TensorSplit: Threads:8 LibrarySearchPath:/tmp/localai/backend_data/backend-assets/gpt4all RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 ControlNet: Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0 Type: FlashAttention:false NoKVOffload:false}
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:46517): stderr load_model: error 'Model format not supported (no matching implementation found)'
9:38PM INF [gpt4all] Fails: could not load model: rpc error: code = Unknown desc = failed loading model
9:38PM INF [llama-cpp-fallback] Attempting to load
9:38PM INF Loading model 'phi-2-layla-v1-chatml-Q8_0.gguf' with backend llama-cpp-fallback
9:38PM DBG Loading model in memory from file: /usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf
9:38PM DBG Loading Model phi-2-layla-v1-chatml-Q8_0.gguf with gRPC (file: /usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf) (backend: llama-cpp-fallback): {backendString:llama-cpp-fallback model:phi-2-layla-v1-chatml-Q8_0.gguf threads:8 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc000af6908 externalBackends:map[] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false parallelRequests:false}
9:38PM DBG ld.so found
9:38PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/lib/ld.so
9:38PM DBG GRPC Service for phi-2-layla-v1-chatml-Q8_0.gguf will be running at: '127.0.0.1:37019'
9:38PM DBG GRPC Service state dir: /tmp/go-processmanager3636492751
9:38PM DBG GRPC Service Started
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:37019): stdout Server listening on 127.0.0.1:37019
9:38PM DBG GRPC Service Ready
9:38PM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:} sizeCache:0 unknownFields:[] Model:phi-2-layla-v1-chatml-Q8_0.gguf ContextSize:4096 Seed:2052651737 NBatch:512 F16Memory:true MLock:false MMap:true VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:99999999 MainGPU: TensorSplit: Threads:8 LibrarySearchPath:/tmp/localai/backend_data/backend-assets/gpt4all RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 ControlNet: Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0 Type: FlashAttention:false NoKVOffload:false}
9:38PM INF [llama-cpp-fallback] Fails: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF
9:38PM INF [rwkv] Attempting to load
9:38PM INF Loading model 'phi-2-layla-v1-chatml-Q8_0.gguf' with backend rwkv
9:38PM DBG Loading model in memory from file: /usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf
9:38PM DBG Loading Model phi-2-layla-v1-chatml-Q8_0.gguf with gRPC (file: /usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf) (backend: rwkv): {backendString:rwkv model:phi-2-layla-v1-chatml-Q8_0.gguf threads:8 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc000af6908 externalBackends:map[] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false parallelRequests:false}
9:38PM DBG ld.so found
9:38PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/lib/ld.so
9:38PM DBG GRPC Service for phi-2-layla-v1-chatml-Q8_0.gguf will be running at: '127.0.0.1:40673'
9:38PM DBG GRPC Service state dir: /tmp/go-processmanager2968480270
9:38PM DBG GRPC Service Started
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:40673): stderr 2024/07/01 21:38:39 gRPC Server listening at 127.0.0.1:40673
9:38PM DBG GRPC Service Ready
9:38PM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:} sizeCache:0 unknownFields:[] Model:phi-2-layla-v1-chatml-Q8_0.gguf ContextSize:4096 Seed:2052651737 NBatch:512 F16Memory:true MLock:false MMap:true VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:99999999 MainGPU: TensorSplit: Threads:8 LibrarySearchPath:/tmp/localai/backend_data/backend-assets/gpt4all RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 ControlNet: Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0 Type: FlashAttention:false NoKVOffload:false}
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:40673): stderr
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:40673): stderr /home/runner/_work/LocalAI/LocalAI/sources/go-rwkv.cpp/rwkv.cpp/rwkv_file_format.inc:93: header.magic == 0x67676d66
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:40673): stderr Invalid file header
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:40673): stderr /home/runner/_work/LocalAI/LocalAI/sources/go-rwkv.cpp/rwkv.cpp/rwkv_model_loading.inc:158: rwkv_fread_file_header(file.file, model.header)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:40673): stderr
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:40673): stderr /home/runner/_work/LocalAI/LocalAI/sources/go-rwkv.cpp/rwkv.cpp/rwkv.cpp:63: rwkv_load_model_from_file(file_path, *ctx->model)
9:38PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:40673): stderr 2024/07/01 21:38:41 InitFromFile /usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf failed
9:38PM INF [rwkv] Fails: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF
9:38PM INF [whisper] Attempting to load
9:38PM INF Loading model 'phi-2-layla-v1-chatml-Q8_0.gguf' with backend whisper
9:38PM DBG Loading model in memory from file: /usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf
9:38PM DBG Loading Model phi-2-layla-v1-chatml-Q8_0.gguf with gRPC (file: /usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf) (backend: whisper): {backendString:whisper model:phi-2-layla-v1-chatml-Q8_0.gguf threads:8 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc000af6908 externalBackends:map[] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false parallelRequests:false}
9:38PM DBG ld.so found
9:38PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/lib/ld.so
9:38PM DBG GRPC Service for phi-2-layla-v1-chatml-Q8_0.gguf will be running at: '127.0.0.1:45435'
9:38PM DBG GRPC Service state dir: /tmp/go-processmanager3232618578
9:38PM DBG GRPC Service Started
9:39PM ERR failed starting/connecting to the gRPC service error="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:45435: connect: connection refused""
9:39PM DBG GRPC Service NOT ready
9:39PM INF [whisper] Fails: grpc service not ready
9:39PM INF [huggingface] Attempting to load
9:39PM INF Loading model 'phi-2-layla-v1-chatml-Q8_0.gguf' with backend huggingface
9:39PM DBG Loading model in memory from file: /usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf
9:39PM DBG Loading Model phi-2-layla-v1-chatml-Q8_0.gguf with gRPC (file: /usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf) (backend: huggingface): {backendString:huggingface model:phi-2-layla-v1-chatml-Q8_0.gguf threads:8 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc000af6908 externalBackends:map[] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false parallelRequests:false}
9:39PM DBG ld.so found
9:39PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/lib/ld.so
9:39PM DBG GRPC Service for phi-2-layla-v1-chatml-Q8_0.gguf will be running at: '127.0.0.1:46715'
9:39PM DBG GRPC Service state dir: /tmp/go-processmanager3136662323
9:39PM DBG GRPC Service Started
9:39PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:46715): stderr 2024/07/01 21:39:21 gRPC Server listening at 127.0.0.1:46715
9:39PM DBG GRPC Service Ready
9:39PM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:} sizeCache:0 unknownFields:[] Model:phi-2-layla-v1-chatml-Q8_0.gguf ContextSize:4096 Seed:2052651737 NBatch:512 F16Memory:true MLock:false MMap:true VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:99999999 MainGPU: TensorSplit: Threads:8 LibrarySearchPath:/tmp/localai/backend_data/backend-assets/gpt4all RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 ControlNet: Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0 Type: FlashAttention:false NoKVOffload:false}
9:39PM INF [huggingface] Fails: could not load model: rpc error: code = Unknown desc = no huggingface token provided
9:39PM INF [bert-embeddings] Attempting to load
9:39PM INF Loading model 'phi-2-layla-v1-chatml-Q8_0.gguf' with backend bert-embeddings
9:39PM DBG Loading model in memory from file: /usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf
9:39PM DBG Loading Model phi-2-layla-v1-chatml-Q8_0.gguf with gRPC (file: /usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf) (backend: bert-embeddings): {backendString:bert-embeddings model:phi-2-layla-v1-chatml-Q8_0.gguf threads:8 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc000af6908 externalBackends:map[] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false parallelRequests:false}
9:39PM DBG ld.so found
9:39PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/lib/ld.so
9:39PM DBG GRPC Service for phi-2-layla-v1-chatml-Q8_0.gguf will be running at: '127.0.0.1:33389'
9:39PM DBG GRPC Service state dir: /tmp/go-processmanager3903548956
9:39PM DBG GRPC Service Started
9:39PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:33389): stderr 2024/07/01 21:39:23 gRPC Server listening at 127.0.0.1:33389
9:39PM DBG GRPC Service Ready
9:39PM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:} sizeCache:0 unknownFields:[] Model:phi-2-layla-v1-chatml-Q8_0.gguf ContextSize:4096 Seed:2052651737 NBatch:512 F16Memory:true MLock:false MMap:true VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:99999999 MainGPU: TensorSplit: Threads:8 LibrarySearchPath:/tmp/localai/backend_data/backend-assets/gpt4all RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 ControlNet: Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0 Type: FlashAttention:false NoKVOffload:false}
9:39PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:33389): stderr bert_load_from_file: invalid model file '/usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf' (bad magic)
9:39PM DBG GRPC(phi-2-layla-v1-chatml-Q8_0.gguf-127.0.0.1:33389): stderr bert_bootstrap: failed to load model from '/usr/share/local-ai/models/phi-2-layla-v1-chatml-Q8_0.gguf'
9:39PM INF [bert-embeddings] Fails: could not load model: rpc error: code = Unknown desc = failed loading model
9:39PM ERR Server error error="could not load model - all backends returned error: [llama-cpp]: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF\n[llama-cpp]: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF\n[llama-ggml]: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF\n[gpt4all]: could not load model: rpc error: code = Unknown desc = failed loading model\n[llama-cpp-fallback]: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF\n[rwkv]: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF\n[whisper]: grpc service not ready\n[huggingface]: could not load model: rpc error: code = Unknown desc = no huggingface token provided\n[bert-embeddings]: could not load model: rpc error: code = Unknown desc = failed loading model" ip=127.0.0.1 latency=56.113362194s method=POST status=500 url=/v1/chat/completions

Additional notes:

1. I tried multiple install configurations and none worked.
2. I start with a fresh VM everytime I did a new setup for localai
3. I tried different models from the localai model page and the results were thr same
4. I settled on the phi-2-chat:Q8_0 since it is in the localai github

Let me know if I need to do something else.

Thanks for the support!

[CyberGWJ]

@mudler Forgot to add. This is the phi-2-chat yaml file.

name: phi-2-chat:Q8_0
mmap: true
context_size: 4096
f16: true
parameters:
model: phi-2-layla-v1-chatml-Q8_0.gguf
stopwords:

• <|im_end|>

template:
chat_message: |
<|im_start|>{{if eq .RoleName "assistant"}}assistant{{else if eq .RoleName "system"}}system{{else if eq .RoleName "user"}}user{{end}}
{{if .Content}}{{.Content}}{{end}}
<|im_end|>
chat: |
{{.Input}}
<|im_start|>assistant
completion: |
{{.Input}}
usage: |
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "phi-2-chat",
"messages": [{"role": "user", "content": "How are you doing?", "temperature": 0.1}]
}'

[CyberGWJ]

@mudler Hers is the latest update. I have tried 3 different versions of linux and 6 different models. They all give the same errors. The only thing I did notice is that during the installation there is an error.

aiadmin@localai:~$ curl https://localai.io/install.sh | PORT=8989 sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 20078 0 20078 0 0 32308 0 --:--:-- --:--:-- --:--:-- 32279
sh: 585: [: Linux: unexpected operator <=====================
Downloading local-ai...
######################################################################## 100.0%
Installing local-ai to /usr/local/bin...
Creating local-ai user...
Adding current user to local-ai group...
Creating local-ai systemd service...
Enabling and starting local-ai service...
Created symlink /etc/systemd/system/default.target.wants/local-ai.service → /etc/systemd/system/local-ai.service.
The LocalAI API is now available at 127.0.0.1:8989.
Install complete. Run "local-ai" from the command line.
[WARN] No NVIDIA/AMD GPU detected. LocalAI will run in CPU-only mode.

I will continue to look at other areas. Although I am running out of ideas.

[CyberGWJ]

@mudler I got it working and you should document this somewhere. I found a clue in this issue #2220 (GRPC service not ready). The issue stated it was fixed but there maybe more that needs to be done.

I had to change the CPU type on the VM and now all the models work.

A notice or warning for those using VMs to insure that the CPU type is correct. Otherwise localai could generate the RPC errors.

[CyberGWJ]

I had to change the CPU type on the VM and now all the models work.

[TheDom42]

Hi, I believe, I have to ressurect both this thread as well as #2220:
I am trying to set up the LocalAI AIO docker container (localai/localai:latest-aio-gpu-intel-f32) on Unraid (6.12.10) with /dev/dri passed for the iGPU of a i5-12600K. The startup seems okay and I see no error messages here, but when I enter a text in the chat UI on the server, the waiting indicator dots do not turn off until eventually it times out:

In the logs, I can see the same debug messages regarding [llama-cpp-fallback] Fails: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF (only partly copied due to length):

Click to extend log

6:09PM INF [llama-cpp-fallback] Fails: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF
6:09PM INF [rwkv] Attempting to load
6:09PM INF Loading model 'b5869d55688a529c3738cb044e92c331' with backend rwkv
6:09PM DBG Loading model in memory from file: /build/models/b5869d55688a529c3738cb044e92c331
6:09PM DBG Loading Model b5869d55688a529c3738cb044e92c331 with gRPC (file: /build/models/b5869d55688a529c3738cb044e92c331) (backend: rwkv): {backendString:rwkv model:b5869d55688a529c3738cb044e92c331 threads:8 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc0001d26c8 externalBackends:map[autogptq:/build/backend/python/autogptq/run.sh bark:/build/backend/python/bark/run.sh coqui:/build/backend/python/coqui/run.sh diffusers:/build/backend/python/diffusers/run.sh exllama:/build/backend/python/exllama/run.sh exllama2:/build/backend/python/exllama2/run.sh huggingface-embeddings:/build/backend/python/sentencetransformers/run.sh mamba:/build/backend/python/mamba/run.sh openvoice:/build/backend/python/openvoice/run.sh parler-tts:/build/backend/python/parler-tts/run.sh petals:/build/backend/python/petals/run.sh rerankers:/build/backend/python/rerankers/run.sh sentencetransformers:/build/backend/python/sentencetransformers/run.sh transformers:/build/backend/python/transformers/run.sh transformers-musicgen:/build/backend/python/transformers-musicgen/run.sh vall-e-x:/build/backend/python/vall-e-x/run.sh vllm:/build/backend/python/vllm/run.sh] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false parallelRequests:false}
6:09PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/grpc/rwkv
6:09PM DBG GRPC Service for b5869d55688a529c3738cb044e92c331 will be running at: '127.0.0.1:35385'
6:09PM DBG GRPC Service state dir: /tmp/go-processmanager3003024065
6:09PM DBG GRPC Service Started
6:09PM DBG GRPC(b5869d55688a529c3738cb044e92c331-127.0.0.1:35385): stderr 2024/07/09 18:09:09 gRPC Server listening at 127.0.0.1:35385
6:09PM DBG GRPC Service Ready
6:09PM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:<nil>} sizeCache:0 unknownFields:[] Model:b5869d55688a529c3738cb044e92c331 ContextSize:8192 Seed:844471670 NBatch:512 F16Memory:false MLock:false MMap:false VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:99999999 MainGPU: TensorSplit: Threads:8 LibrarySearchPath:/tmp/localai/backend_data/backend-assets/gpt4all RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/build/models/b5869d55688a529c3738cb044e92c331 Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 ControlNet: Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0 Type: FlashAttention:false NoKVOffload:false}
6:09PM DBG GRPC(b5869d55688a529c3738cb044e92c331-127.0.0.1:35385): stderr 
6:09PM DBG GRPC(b5869d55688a529c3738cb044e92c331-127.0.0.1:35385): stderr /build/sources/go-rwkv.cpp/rwkv.cpp/rwkv_file_format.inc:93: header.magic == 0x67676d66
6:09PM DBG GRPC(b5869d55688a529c3738cb044e92c331-127.0.0.1:35385): stderr Invalid file header
6:09PM DBG GRPC(b5869d55688a529c3738cb044e92c331-127.0.0.1:35385): stderr /build/sources/go-rwkv.cpp/rwkv.cpp/rwkv_model_loading.inc:158: rwkv_fread_file_header(file.file, model.header)
6:09PM DBG GRPC(b5869d55688a529c3738cb044e92c331-127.0.0.1:35385): stderr 
6:09PM DBG GRPC(b5869d55688a529c3738cb044e92c331-127.0.0.1:35385): stderr /build/sources/go-rwkv.cpp/rwkv.cpp/rwkv.cpp:63: rwkv_load_model_from_file(file_path, *ctx->model)
6:09PM DBG GRPC(b5869d55688a529c3738cb044e92c331-127.0.0.1:35385): stderr 2024/07/09 18:09:11 InitFromFile /build/models/b5869d55688a529c3738cb044e92c331 failed
6:09PM INF [rwkv] Fails: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF
6:09PM INF [whisper] Attempting to load
6:09PM INF Loading model 'b5869d55688a529c3738cb044e92c331' with backend whisper
6:09PM DBG Loading model in memory from file: /build/models/b5869d55688a529c3738cb044e92c331
6:09PM DBG Loading Model b5869d55688a529c3738cb044e92c331 with gRPC (file: /build/models/b5869d55688a529c3738cb044e92c331) (backend: whisper): {backendString:whisper model:b5869d55688a529c3738cb044e92c331 threads:8 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc0001d26c8 externalBackends:map[autogptq:/build/backend/python/autogptq/run.sh bark:/build/backend/python/bark/run.sh coqui:/build/backend/python/coqui/run.sh diffusers:/build/backend/python/diffusers/run.sh exllama:/build/backend/python/exllama/run.sh exllama2:/build/backend/python/exllama2/run.sh huggingface-embeddings:/build/backend/python/sentencetransformers/run.sh mamba:/build/backend/python/mamba/run.sh openvoice:/build/backend/python/openvoice/run.sh parler-tts:/build/backend/python/parler-tts/run.sh petals:/build/backend/python/petals/run.sh rerankers:/build/backend/python/rerankers/run.sh sentencetransformers:/build/backend/python/sentencetransformers/run.sh transformers:/build/backend/python/transformers/run.sh transformers-musicgen:/build/backend/python/transformers-musicgen/run.sh vall-e-x:/build/backend/python/vall-e-x/run.sh vllm:/build/backend/python/vllm/run.sh] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false parallelRequests:false}
6:09PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/grpc/whisper
6:09PM DBG GRPC Service for b5869d55688a529c3738cb044e92c331 will be running at: '127.0.0.1:33985'
6:09PM DBG GRPC Service state dir: /tmp/go-processmanager4043782667
6:09PM DBG GRPC Service Started
6:09PM DBG GRPC(b5869d55688a529c3738cb044e92c331-127.0.0.1:33985): stderr 2024/07/09 18:09:11 gRPC Server listening at 127.0.0.1:33985
6:09PM DBG GRPC Service Ready
6:09PM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:<nil>} sizeCache:0 unknownFields:[] Model:b5869d55688a529c3738cb044e92c331 ContextSize:8192 Seed:844471670 NBatch:512 F16Memory:false MLock:false MMap:false VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:99999999 MainGPU: TensorSplit: Threads:8 LibrarySearchPath:/tmp/localai/backend_data/backend-assets/gpt4all RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/build/models/b5869d55688a529c3738cb044e92c331 Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 ControlNet: Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0 Type: FlashAttention:false NoKVOffload:false}
6:09PM DBG GRPC(b5869d55688a529c3738cb044e92c331-127.0.0.1:33985): stderr whisper_init_from_file_with_params_no_state: loading model from '/build/models/b5869d55688a529c3738cb044e92c331'
6:09PM DBG GRPC(b5869d55688a529c3738cb044e92c331-127.0.0.1:33985): stderr whisper_init_with_params_no_state: use gpu    = 1
6:09PM DBG GRPC(b5869d55688a529c3738cb044e92c331-127.0.0.1:33985): stderr whisper_init_with_params_no_state: flash attn = 0
6:09PM DBG GRPC(b5869d55688a529c3738cb044e92c331-127.0.0.1:33985): stderr whisper_init_with_params_no_state: gpu_device = 0
6:09PM DBG GRPC(b5869d55688a529c3738cb044e92c331-127.0.0.1:33985): stderr whisper_init_with_params_no_state: dtw        = 0
6:09PM DBG GRPC(b5869d55688a529c3738cb044e92c331-127.0.0.1:33985): stderr whisper_model_load: loading model
6:09PM DBG GRPC(b5869d55688a529c3738cb044e92c331-127.0.0.1:33985): stderr whisper_model_load: invalid model data (bad magic)
6:09PM DBG GRPC(b5869d55688a529c3738cb044e92c331-127.0.0.1:33985): stderr whisper_init_with_params_no_state: failed to load model
6:09PM INF [whisper] Fails: could not load model: rpc error: code = Unknown desc = unable to load model
6:09PM INF [piper] Attempting to load
6:09PM INF Loading model 'b5869d55688a529c3738cb044e92c331' with backend piper
6:09PM DBG Loading model in memory from file: /build/models/b5869d55688a529c3738cb044e92c331
6:09PM DBG Loading Model b5869d55688a529c3738cb044e92c331 with gRPC (file: /build/models/b5869d55688a529c3738cb044e92c331) (backend: piper): {backendString:piper model:b5869d55688a529c3738cb044e92c331 threads:8 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc0001d26c8 externalBackends:map[autogptq:/build/backend/python/autogptq/run.sh bark:/build/backend/python/bark/run.sh coqui:/build/backend/python/coqui/run.sh diffusers:/build/backend/python/diffusers/run.sh exllama:/build/backend/python/exllama/run.sh exllama2:/build/backend/python/exllama2/run.sh huggingface-embeddings:/build/backend/python/sentencetransformers/run.sh mamba:/build/backend/python/mamba/run.sh openvoice:/build/backend/python/openvoice/run.sh parler-tts:/build/backend/python/parler-tts/run.sh petals:/build/backend/python/petals/run.sh rerankers:/build/backend/python/rerankers/run.sh sentencetransformers:/build/backend/python/sentencetransformers/run.sh transformers:/build/backend/python/transformers/run.sh transformers-musicgen:/build/backend/python/transformers-musicgen/run.sh vall-e-x:/build/backend/python/vall-e-x/run.sh vllm:/build/backend/python/vllm/run.sh] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false parallelRequests:false}
6:09PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/grpc/piper
6:09PM DBG GRPC Service for b5869d55688a529c3738cb044e92c331 will be running at: '127.0.0.1:42911'
6:09PM DBG GRPC Service state dir: /tmp/go-processmanager1824935975
6:09PM DBG GRPC Service Started
6:09PM DBG GRPC(b5869d55688a529c3738cb044e92c331-127.0.0.1:42911): stderr 2024/07/09 18:09:13 gRPC Server listening at 127.0.0.1:42911
6:09PM DBG GRPC Service Ready
6:09PM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:<nil>} sizeCache:0 unknownFields:[] Model:b5869d55688a529c3738cb044e92c331 ContextSize:8192 Seed:844471670 NBatch:512 F16Memory:false MLock:false MMap:false VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:99999999 MainGPU: TensorSplit: Threads:8 LibrarySearchPath:/tmp/localai/backend_data/backend-assets/espeak-ng-data RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/build/models/b5869d55688a529c3738cb044e92c331 Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 ControlNet: Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0 Type: FlashAttention:false NoKVOffload:false}
6:09PM INF [piper] Fails: could not load model: rpc error: code = Unknown desc = unsupported model type /build/models/b5869d55688a529c3738cb044e92c331 (should end with .onnx)
6:09PM INF [stablediffusion] Attempting to load
6:09PM INF Loading model 'b5869d55688a529c3738cb044e92c331' with backend stablediffusion
6:09PM DBG Loading model in memory from file: /build/models/b5869d55688a529c3738cb044e92c331
6:09PM DBG Loading Model b5869d55688a529c3738cb044e92c331 with gRPC (file: /build/models/b5869d55688a529c3738cb044e92c331) (backend: stablediffusion): {backendString:stablediffusion model:b5869d55688a529c3738cb044e92c331 threads:8 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc0001d26c8 externalBackends:map[autogptq:/build/backend/python/autogptq/run.sh bark:/build/backend/python/bark/run.sh coqui:/build/backend/python/coqui/run.sh diffusers:/build/backend/python/diffusers/run.sh exllama:/build/backend/python/exllama/run.sh exllama2:/build/backend/python/exllama2/run.sh huggingface-embeddings:/build/backend/python/sentencetransformers/run.sh mamba:/build/backend/python/mamba/run.sh openvoice:/build/backend/python/openvoice/run.sh parler-tts:/build/backend/python/parler-tts/run.sh petals:/build/backend/python/petals/run.sh rerankers:/build/backend/python/rerankers/run.sh sentencetransformers:/build/backend/python/sentencetransformers/run.sh transformers:/build/backend/python/transformers/run.sh transformers-musicgen:/build/backend/python/transformers-musicgen/run.sh vall-e-x:/build/backend/python/vall-e-x/run.sh vllm:/build/backend/python/vllm/run.sh] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false parallelRequests:false}
6:09PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/grpc/stablediffusion
6:09PM DBG GRPC Service for b5869d55688a529c3738cb044e92c331 will be running at: '127.0.0.1:33259'
6:09PM DBG GRPC Service state dir: /tmp/go-processmanager1780102731
6:09PM DBG GRPC Service Started
6:09PM DBG GRPC(b5869d55688a529c3738cb044e92c331-127.0.0.1:33259): stderr 2024/07/09 18:09:15 gRPC Server listening at 127.0.0.1:33259
6:09PM DBG GRPC Service Ready
6:09PM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:<nil>} sizeCache:0 unknownFields:[] Model:b5869d55688a529c3738cb044e92c331 ContextSize:8192 Seed:844471670 NBatch:512 F16Memory:false MLock:false MMap:false VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:99999999 MainGPU: TensorSplit: Threads:8 LibrarySearchPath:/tmp/localai/backend_data/backend-assets/espeak-ng-data RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/build/models/b5869d55688a529c3738cb044e92c331 Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 ControlNet: Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 TensorParallelSize:0 MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0 Type: FlashAttention:false NoKVOffload:false}
6:09PM INF [stablediffusion] Loads OK

I have figured out that this might be due to problems in recognising the AVX capabilities of the server CPU. However, I would not know how to change the CPU type for a docker container, that is NOT in a VM. Maybe the fix in #2220 and #2306 is not complete?

[rickardrockard]

Same issue here using Docker on Unraid.