Issue loading Metal kernels from shared library

[abetlen]

Just looking for some help on this issue integrating the new Metal support as I'm not really familiar with MacOS.

The issue, orignally reported here, seems to be that path resolution for the ggml-metal.metal file fails when llama.cpp is built as a shared library. This produces the following output

llama_model_load_internal: mem required  = 2532.67 MB (+ 3124.00 MB per state)
....................................................................................................
llama_init_from_file: kv self size  = 3120.00 MB
ggml_metal_init: allocating
ggml_metal_init: using MPS
ggml_metal_init: loading '(null)'
ggml_metal_init: error: Error Domain=NSCocoaErrorDomain Code=258 "The file name is invalid."

It seems the issue stems from ggml_metal_init calling [[NSBundle mainBundle] pathForResource:@"ggml-metal" ofType:@"metal"] to locate the ggml-metal.metal file.

I've tried copying the file to a location relative to the shared library but that doesn't seem to work. It seems that pathForResource expects the file location relative to the absolute path of the executable loading the shared libray (in this case system python).

If anyone here has any thoughts please let me know, much appreciated.

[j-f1]

I think this can be fixed by doing something like:

@interface BundleHook
@end
@implementation BundleHook
@end

NSBundle *bundle = [NSBundle bundleForClass:[BundleHook class]];

[swittk]

Using bundleForClass is a good hack, I can confirm it works for if the .metal file is placed in the same directory as the dylib.
(added the following in ggml-metal.m)

// Here to assist with NSBundle Path Hack
@interface GGMLMetalClass : NSObject
@end
@implementation GGMLMetalClass
@end
...
        NSBundle *bundle = [NSBundle bundleForClass:[GGMLMetalClass class]];
        NSString *bundlePath = [bundle bundlePath];
        NSString * path = [bundle pathForResource:@"ggml-metal" ofType:@"metal"];
        NSLog(@"Bundle is %@; at path %@, the path for resource is %@", bundle, bundlePath, path);
...

You'd get the following : Bundle is NSBundle </Users/username/Downloads/llama-cpp-python/llama_cpp> (loaded); at path /Users/username/Downloads/llama-cpp-python/llama_cpp, the path for resource is /Users/username/Downloads/llama-cpp-python/llama_cpp/ggml-metal.metal
The metal shader can be found, loaded, and run with this method. The less hacky (but messy) way might be to add the whole shader as an imported string (like we're doing with OpenCL).