MPS Support - CMAKE_ARGS="LLAMA_METAL=1"

[leedrake5]

The main llama.cpp has been updated to support GPUs on Mac's with the following flag (tested on my system):

LLAMA_METAL=1 make -j && ./main -m /Downloads/guanaco-65B.ggmlv3.q4_0.bin -p "I believe the meaning of life is" --ignore-eos -n 64 -ngl 1

It look like the following flag needs to be added to CMake options:

CMAKE_ARGS="LLAMA_METAL=1" FORCE_CMAKE=1 pip install -e .

While it appears that it installs successfully, the library cannot be loaded.

>>> from llama_cpp import Llama, LlamaCache
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/llama_cpp/__init__.py", line 1, in <module>
    from .llama_cpp import *
  File "/opt/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/llama_cpp/llama_cpp.py", line 73, in <module>
    _lib = _load_shared_library(_lib_base_name)
  File "/opt/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/llama_cpp/llama_cpp.py", line 64, in _load_shared_library
    raise FileNotFoundError(
FileNotFoundError: Shared library with base name 'llama' not found

This happens regardless of whether the GitHub repo or PyPy are used.

[zach-brockway]

I was able to patch this locally:

# Load the library
def _load_shared_library(lib_base_name: str):
    # Determine the file extension based on the platform
    if sys.platform.startswith("linux"):
        lib_ext = ".so"
    elif sys.platform == "darwin":
        lib_ext = ".dylib" # <<< Was also ".so"

However, I still seem to get crashes trying to load models subsequently:

llama_model_load_internal: mem required  = 2532.67 MB (+ 3124.00 MB per state)
....................................................................................................
llama_init_from_file: kv self size  = 3120.00 MB
ggml_metal_init: allocating
ggml_metal_init: using MPS
ggml_metal_init: loading '(null)'
ggml_metal_init: error: Error Domain=NSCocoaErrorDomain Code=258 "The file name is invalid."
/Users/voltrondata/github-actions-runner/_work/crossbow/crossbow/arrow/cpp/src/arrow/filesystem/s3fs.cc:2598:  arrow::fs::FinalizeS3 was not called even though S3 was initialized.  This could lead to a segmentation fault at exit

---

Edit: Best I can tell, it's failing to load the Metal shader for some reason, and it seems like that's supposed to be embedded into the dylib somehow?

    // read the source from "ggml-metal.metal" into a string and use newLibraryWithSource
    {
        NSError * error = nil;

        //NSString * path = [[NSBundle mainBundle] pathForResource:@"../../examples/metal/metal" ofType:@"metal"];
        NSString * path = [[NSBundle mainBundle] pathForResource:@"ggml-metal" ofType:@"metal"];
        fprintf(stderr, "%s: loading '%s'\n", __func__, [path UTF8String]);

        NSString * src  = [NSString stringWithContentsOfFile:path encoding:NSUTF8StringEncoding error:&error];
        if (error) {
            fprintf(stderr, "%s: error: %s\n", __func__, [[error description] UTF8String]);
            exit(1);
        }

[zach-brockway]

"Fixed" the second issue by copying llama.cpp/ggml-metal.metal to the same directory as my python binary!

[zach-brockway]

So it seems like the upstream llama.cpp Makefile and CMakeLists disagree about what the extension of the shared library should be. Per this discussion, you can force libllama to be generated with the .so extension instead of .dylib by adding the MODULE keyword here:

add_library(llama MODULE
            llama.cpp
            llama.h
            llama-util.h
            )

Not clear to me if this might negatively impact other platforms, but it's enough to make FORCE_CMAKE builds generate the expected libllama.so, rather than a libllama.dylib that the Python has trouble finding.

[leedrake5]

Thanks @zach-brockway, I can successfully get it to load with this bit:

# Load the library
def _load_shared_library(lib_base_name: str):
    # Determine the file extension based on the platform
    if sys.platform.startswith("linux"):
        lib_ext = ".so"
    elif sys.platform == "darwin":
        lib_ext = ".dylib" # <<< Was also ".so"

This goes into llama_cpp.py in the site-packages folder. However, it still only uses CPU, not GPU, even when I copy llama.cpp/ggml-metal.metal to site-packages/llama_cpp. I suspect it's because I don't know where this is suppose to go:

    // read the source from "ggml-metal.metal" into a string and use newLibraryWithSource
    {
        NSError * error = nil;

        //NSString * path = [[NSBundle mainBundle] pathForResource:@"../../examples/metal/metal" ofType:@"metal"];
        NSString * path = [[NSBundle mainBundle] pathForResource:@"ggml-metal" ofType:@"metal"];
        fprintf(stderr, "%s: loading '%s'\n", __func__, [path UTF8String]);

        NSString * src  = [NSString stringWithContentsOfFile:path encoding:NSUTF8StringEncoding error:&error];
        if (error) {
            fprintf(stderr, "%s: error: %s\n", __func__, [[error description] UTF8String]);
            exit(1);
        }

Also where does this function go?

add_library(llama MODULE
            llama.cpp
            llama.h
            llama-util.h
            )

[zach-brockway]

Also where does this function go?

add_library(llama MODULE
            llama.cpp
            llama.h
            llama-util.h
            )

That was a modification I had to make to vendor/llama.cpp/CMakeLists.txt (line 412). The workarounds are starting to pile up at every level! But I'm pretty sure llama.cpp will in fact want to take a fix for this upstream. Either their static Makefile should output .dylib, or their CMakeLists.txt should output .so; no good reason for the current discrepancy.

[leedrake5]

Many thanks, but worried this may be a dead end. If I use this version in CMakeLists.txt:

add_library(llama MODULE
            llama.cpp
            llama.h
            llama-util.h
            )

I get this error:

CMake Error at CMakeLists.txt:418 (target_include_directories):
  Cannot specify include directories for target "llama" which is not built by
  this project.
CMake Error at tests/CMakeLists.txt:4 (target_link_libraries):
  Target "llama" of type MODULE_LIBRARY may not be linked into another
  target.  One may link only to INTERFACE, OBJECT, STATIC or SHARED
  libraries, or to executables with the ENABLE_EXPORTS property set.
Call Stack (most recent call first):
  tests/CMakeLists.txt:9 (llama_add_test)

If I try to make it more explicit:

add_library(llama.so
            llama.cpp
            llama.h
            llama-util.h
            )

I get the same error. I really appreciate your help trying to workaround this, but I think you are right, this needs to happen upstream. It works fine by command line, but interfacing with the python package makes it very difficult.

[abetlen]

Sorry for the slow reply, I should be able to get access to an M1 tonight and get this sorted, cheers.

[mhenrichsen]

@abetlen sounds awesome. Please let me know if you're having issues and I'll let you ssh into one of mine :)

[Jchang4]

@abetlen hey any updates? This would be an amazing update!

[fungyeung]

"Fixed" the second issue by copying llama.cpp/ggml-metal.metal to the same directory as my python binary!

I tried copying ggml-metal.metal to multiple locations but still got the "file name is invalid" error.
Eventually, I "fixed" it by hardcoding the absolute path in vendor/llama.cpp/ggml-metal.m around line 101:

//NSString * path = [[NSBundle mainBundle] pathForResource:@"ggml-metal" ofType:@"metal"];
NSString * path = @"/path/to/vendor/llama.cpp/ggml-metal.metal";

Then recompile it.

[abetlen]

I added an option to llama_cpp.py to accept both .so and .dylib extensions on macos.

"Fixed" the second issue by copying llama.cpp/ggml-metal.metal to the same directory as my python binary!

@zach-brockway can you expand on this?