After b1177 it failed load gguf model on Intel macOS 13.5.1 #macos

[mayulu]

Hi,
From couple days(maybe 7~8) ago, latest build kept failed loading gguf models.
Before that it worked very well.
I followed instructions like https://github.com/ggerganov/llama.cpp/issues/1730#issuecomment-1636055251, no help.
when run main, the error meaage:

$bin git:(master) ./main -m /Users/mayulu/Documents/llama-2-7b.ggmlv3.q4_K_M.gguf -p "Hello"

Log start
main: build = 1207 (ec2a24f)
main: seed  = 1694227613
llama_model_loader: loaded meta data with 19 key-value pairs and 291 tensors from /Users/mayulu/Documents/llama-2-7b.ggmlv3.q4_K_M.gguf (version GGUF V2 (latest))
llama_model_loader: - tensor    0:                token_embd.weight q4_K     [  4096, 32000,     1,     1 ]

.......

ggml_metal_init: loaded kernel_mul_mat_q5_K_f32            0x7fcdf8c1ba90 | th_max =  512 | th_width =   32
ggml_metal_init: loaded kernel_mul_mat_q6_K_f32            0x7fce09106b10 | th_max = 1024 | th_width =   32
ggml_metal_init: loaded kernel_mul_mm_f16_f32                         0x0 | th_max =    0 | th_width =    0
ggml_metal_init: load pipeline error: Error Domain=CompilerError Code=2 "SC compilation failure
There is a call to an undefined label" UserInfo={NSLocalizedDescription=SC compilation failure
There is a call to an undefined label}
llama_new_context_with_model: ggml_metal_init() failed
llama_init_from_gpt_params: error: failed to create context with model '/Users/mayulu/Documents/GPT4ALL/llama-2-7b.ggmlv3.q4_K_M.gguf'
main: error: unable to load model

ENV: Intel macOS 13.5.1, llama-cpp-python 0.1.83.

would you give some clue? @ggerganov Thanks.

[KerfuffleV2]

It would be helpful if you could tell us the exact build when it stopped working.

[mayulu]

I am afraid I can't figure out the exact build.
After llama.cpp support gguf, I built via cmake following the instruction, it worked very well.
And then I did not keep updating, until couple days ago, when I update to the latest build, it keeps fail.

[KerfuffleV2]

I am afraid I can't figure out the exact build.

Sure you can.

You said the problem started about 7-8 days ago, so try a build from 7-8 days ago. If it doesn't have the problem, try one from a day or two later. If it does, try one from a few days earlier. You can repeat that process until you know exactly what build it stopped working at.

[mayulu]

b1176 and before, it works very well.
After b1177 it fails.
Thank a lot! @KerfuffleV2
ENV: Intel macOS 13.5.1, AMD 5500m, llama-cpp-python 0.1.83.

[KerfuffleV2]

One thing that seems to have changed between b1176 and b1177 is enabling GPU inference by default. I'm not a Mac person but hopefully now there's enough information for someone else to help you if I can't.

This is just a guess, but I think before that build you might just have been compiling without Metal at all. If that is the case, you can try setting LLAMA_NO_METAL=1. One way to verify that is to look at the output from running with b1176 and see if the output seems to indicate that it's using metal. You might also just be able to do -ngl 0 to turn off GPU inference, I'm not sure if that will work/have the same result.

I don't know if Metal is actually supposed to work on Intel Macs.

[mayulu]

On b1177 cmake -DLLAMA_METAL=OFF .. it works. And cmake -DLLAMA_METAL=ON .. it fail.
On b1176 and before I alway use cmake -DLLAMA_METAL=ON .., it always worked. seems It support metal of Intel macOS.
So I guess maybe b1177 and after changed other more about Metal except Metal inference by default. @KerfuffleV2

[KerfuffleV2]

Does the performance seem about the same as before (when you compile with LLAMA_METAL=OFF)? If so that most likely means it just wasn't enabled before.

[mayulu]

b1176 inference log with LLAMA_METAL=ON :

llama_print_timings:        load time =  1727.02 ms
llama_print_timings:      sample time =    89.97 ms /   140 runs   (    0.64 ms per token,  1556.09 tokens per second)
llama_print_timings: prompt eval time =   170.17 ms /     2 tokens (   85.09 ms per token,    11.75 tokens per second)
llama_print_timings:        eval time = 20897.13 ms /   139 runs   (  150.34 ms per token,     6.65 tokens per second)
llama_print_timings:       total time = 21205.78 ms

b1177 inference log with LLAMA_METAL=OFF :

llama_print_timings:        load time =  2125.87 ms
llama_print_timings:      sample time =   134.97 ms /   211 runs   (    0.64 ms per token,  1563.30 tokens per second)
llama_print_timings: prompt eval time =   832.35 ms /     2 tokens (  416.18 ms per token,     2.40 tokens per second)
llama_print_timings:        eval time = 33428.40 ms /   210 runs   (  159.18 ms per token,     6.28 tokens per second)
llama_print_timings:       total time = 34473.05 ms

Eye feels like not too much difference.

But b1176 succeed in loading the model, b1177 failed loading. all with LLAMA_METAL=ON

[KerfuffleV2]

Okay, makes sense. The difference is just that LLAMA_METAL was defaulting to OFF for you before. It got changed to default to ON, but it doesn't seem to work for your configuration.

I don't know if it actually should work in this case, but if you set LLAMA_METAL=OFF you won't be worse off than you were prior to that change.

[mayulu]

Oh, Maybe you are right.
I just found even b1176, when I run with -ngl 1

./main -m /Users/mayulu/Documents/llama-2-7b.ggmlv3.q4_K_M.gguf -p "How about Nim programming language?" -ngl 1

It also failed load the model:

ggml_metal_init: loaded kernel_mul_mat_q6_K_f32            0x7ff6c411c160 | th_max = 1024 | th_width =   32
ggml_metal_init: loaded kernel_mul_mm_f16_f32                         0x0 | th_max =    0 | th_width =    0
ggml_metal_init: load pipeline error: Error Domain=CompilerError Code=2 "SC compilation failure
There is a call to an undefined label" UserInfo={NSLocalizedDescription=SC compilation failure
There is a call to an undefined label}
llama_new_context_with_model: ggml_metal_init() failed
llama_init_from_gpt_params: error: failed to create context with model '/Users/mayulu/Documents/GPT4ALL/llama-2-7b.ggmlv3.q4_K_M.gguf'
main: error: unable to load model

I never use the -ngl 1 option before.
So, maybe llama.cpp never support Intel macOS Metal.
It maybe alway uses the CPU inference. @KerfuffleV2

[KerfuffleV2]

It maybe alway uses the CPU inference.

Yes, that's what I was saying. :)

Whether it's actually correct behavior or not, I don't know. #2407 seems like it may be related.

[mayulu]

Thank you for your time and excellent instructions, I will close this issue. wish you a good day. 👍 @KerfuffleV2