violent crash on Mac Mini M2 8GB RAM when trying to use GPU

[siddhsql]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [ x] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• [ x] I carefully followed the README.md.
• [ x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• [ x] I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

I have a M2 Mac Mini with 8 GB unified memory. I tried to run llama.cpp as explained here: #1642

Current Behavior

My computer froze and rebooted after sometime. I got a brief flash of a pink screen of death. I re-tried several times and got the same behavior. Once instead of crashing, I got an assert in following code:

for (int i = 0; i < n_cb; i++) {
        MTLCommandBufferStatus status = (MTLCommandBufferStatus) [command_buffers[i] status];
        if (status != MTLCommandBufferStatusCompleted) {
            fprintf(stderr, "%s: command buffer %d failed with status %lu\n", __func__, i, status);
            GGML_ASSERT(false);
        }
    }

Each time I was able to see console output saying its trying to load GPU buffers similar to what we see in the video on #1642.

The model I was trying is gpt4-x-vicuna-13B.ggmlv3.q5_K_M.bin

MODEL=gpt4-x-vicuna-13B.ggmlv3.q5_K_M.bin
CONTEXT_SIZE=2048
PROMPT="$SCRIPT_DIR/../prompts/chat-with-bob.txt"

cd $SCRIPT_DIR/../build/release/bin
set -x
./main \
-m "$MODEL" \
 -c $CONTEXT_SIZE \
 --repeat_penalty 1.0 \
--color \
-i \
-r "User:" \
--in-prefix " " \
-f "$PROMPT"

over here I see a funny comment:

yes - this is fixed now that this crashes instead of giving bad output

how is crashing acceptable behaviour?

Environment and Context

Please provide detailed information about your computer setup. This is important in case the issue is not reproducible except for under certain specific conditions.

• Physical (or virtual) hardware you are using, e.g. for Linux:

M2 Mac Mini w/ 8 GB memory

• Operating System, e.g. for Linux:

± uname -a
Darwin 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun  8 22:21:34 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T8112 arm64

• SDK version, e.g. for Linux:

$ python3 --version
$ make --version
$ g++ --version

Failure Information (for bugs)

Please help provide information about the failure if this is a bug. If it is not a bug, please remove the rest of this template.

Steps to Reproduce

see above.

Failure Logs

[BarfingLemurs]

@siddhsql you have a simple problem, you dont have enough ram (only 8gb) to run 13b models. see the sizes in the readme. you can run a 7b model fine.

[siddhsql]

yes i understand that, thanks for the reply. but the program should not crash violently like it did. it should abort but the computer choking up and rebooting is not a good experience. i am afraid to run it on even 7B now.

…

On Fri, Jul 7, 2023 at 7:43 PM BarfingLemurs ***@***.***> wrote: @siddhsql <https://github.com/siddhsql> you have a simple problem, you dont have enough ram (only 8gb) to run 13b models. see the sizes in the readme. you can run a 7b model fine. — Reply to this email directly, view it on GitHub <#2141 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A6NWEKZCN3JNRIBO7M3UY7DXPDCOFANCNFSM6AAAAAA2CJ72GA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[philipturner]

Whenever there's an infinite loop, this behavior is typical of Apple Mac GPUs. The GPU is frozen and won't exit the command. After you reset the computer nothing is broken, looks much scarier than it is.

[siddhsql]

where is the infinite loop in this case?

…

On Tue, Aug 1, 2023 at 7:58 PM Philip Turner ***@***.***> wrote: Whenever there's an infinite loop, this behavior is typical of Apple Mac GPUs. The GPU is frozen and won't exit the command. After you reset the computer nothing is broken, looks much scarier than it is. — Reply to this email directly, view it on GitHub <#2141 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A6NWEK7ZMM636MWU2W636LLXTG66LANCNFSM6AAAAAA2CJ72GA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[philipturner]

It doesn't have to be an infinite loop. Sometimes, giving it a compute workload too heavy causes the GPU to go rogue as well. Often there is some kind of fault that happened internally, for example an out-of-bounds memory access.

On iOS, the GPU goes rogue less often, because a watchdog aborts very long command buffers. On Mac, you have to restart the entire computer. Mac is probably this way to give more flexibility (don't have to actively check whether all your command buffers will fall under 100 ms).

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.