Feature Request: Switch backends dynamically at runtime?

[BrainSlugs83]

Right now, to switch backends, I have to uninstall one nuget package, and then reinstall another. -- This means that if I make an app, I would have to create different builds for every configuration.

Instead I'd rather just give someone a dropdown box in a UI and let them pick the backend, and have the app switch on the fly to whatever they have selected.

Ideally this would just be an enum in the ModelParams, something like Backend = LLamaSharp.Cuda, or Backend = LLamaSharp.Cpu, etc. (Heck, even if it has to be a global static parameter, that's better than having to make separate builds.)

One possible way it could be implemented is by shipping all of the libllama.dll files in the output directory with a slightly different name (i.e. libllama-cuda.dll, libllama-avx.dll, etc.) and dynamically loading the DLL into memory based on which backend is being requested.

[martindevans]

It sounds like there's two parts to this request:

• Ability to configure the backend before it is loaded.
  • At the moment this isn't possible, because the DLL loading is in the static constructor for NativeApi, so the instant you access that class the DLL is loaded. We could introduce another class (e.g. NativeApiConfig) with static config methods/properties on it.
• Ability to unload the backend and load another.
  • This could probably be done by using NativeLibrary.Load and NativeLibrary.Free (docs).

[BrainSlugs83]

It looks like you're already trying to load them one at a time in the static API... what would be the harm in loading all of the ones that are found? and just putting the IntPtrs into an array.

We could set the one we want to use via a static enum property -- and the IntPtr could just be a read only property that returns the appropriate IntPtr based on which enum is selected. (Or throws an exception if a missing one was selected.)

We could have another static method that is just like IEnumerable<Backend> AvailableBackends { get; } that does a yield return Backend.xyz if that backend is available...

That would allow hotswap of the dlls at runtime.

Edit: I suppose you wouldn't want the backend to switch out from under any already created LLMs, or whatever... but even then... you could still load them all into an array and use an enum (just don't do the static property stuff). Just pass the enum in as a model param, and it grabs the backend pointer at creation time.

That would allow you to use multiple backends at the same time.

[martindevans]

Loading all of the backends at once wouldn't work with the current system because the native methods are written like this:

[DllImport("libllama")]
public static extern void demo_method();

That will use the already loaded libllama.dll (if there is one) otherwise it will find and load it when called. Given that I'm not sure exactly what would happen it you loaded multiple, but I'm sure it wouldn't be good!

If we just wanted to allow unloading one backend DLL and loading another I think that could probably be done by keeping the IntPtr returned from NativeLibrary.Load and passing it into NativeLibrary.Free later (currently we don't keep that pointer because never unload libraries). In this case no rewriting of the native symbols would be required because there's only ever one loaded library.

If we wanted to allow multiple backends that's a lot more complex. I'm not even 100% sure if it's possible, is it guaranteed that two versions of libllama.dll don't try to use some per-process state internally? Assuming it is we'd probably have to define a non-static "LLamaBackend" object and then fetch all of the native methods using GetExport on a specific backend. Of course we'd probably also want to allow some kind of default backend, so users can continue using the library roughly as-is without having to create a backend object.

If you're interested in experimenting with any of this stuff I'm happy to help with that. Some prototypes in a fork showing some proof-of-concept unloading/reloading/multi-loading would be great!

P.S. one extra complexity I just thought of: NativeLibrary doesn't exist in NETSTANDARD2_0, so there's no cross platform way to do all this stuff there. It can be done, but you need to call the approprate OS methods directly.

[saddam213]

What if we wrote a bunch of delegates as a sort of wrapped API, and use LoadLibrary to swap dll's out during runtime

However the performance hit could be heavy, also cleanup could become troublesome

    static class KernelNativeMethods
    {
        [DllImport("kernel32.dll")]
        public static extern IntPtr LoadLibrary(string dllToLoad);

        [DllImport("kernel32.dll")]
        public static extern IntPtr GetProcAddress(IntPtr hModule, string procedureName);

        [DllImport("kernel32.dll")]
        public static extern bool FreeLibrary(IntPtr hModule);
    }

    class Program
    {
        [UnmanagedFunctionPointer(CallingConvention.Cdecl)]
        private delegate bool LLamaEmptyCall();

        static void Main(string[] args)
        {
            IntPtr libllama = KernelNativeMethods.LoadLibrary(@"libllama-cuda12.dll");
 
            IntPtr functionToCallAddress = KernelNativeMethods.GetProcAddress(libllama, "llama_mmap_supported");

            LLamaEmptyCall llama_empty_call = (LLamaEmptyCall)Marshal.GetDelegateForFunctionPointer(functionToCallAddress, typeof(LLamaEmptyCall));

            bool result = llama_empty_call();

            Console.WriteLine(result);

            // Free
            KernelNativeMethods.FreeLibrary(libllama);
        }
    }

[martindevans]

What if we wrote a bunch of delegates as a sort of wrapped API, and use LoadLibrary to swap dll's out during runtime

As I understand it that's roughly equivalent to what I was suggesting with NativeLibrary.Load (cross platform equivalent of kernel32.LoadLibrary and NativeLibrary.GetExport (cross platform equivalent of kernel32.GetProcAddress).

A few problems that I'm not sure about with loading multiple backends:

• The backend would have to keep an outgoing weak reference to everything created with it, so that when the backend is destroyed it can first destroy all resources. Not impossible, but complex.
• Is it legal to have multiple versions of llama.cpp loaded at once? e.g. is it possible that they could be using some per-process global state that would break in subtle ways?
• Do we actually want to go as far as supporting multiple backends, or is just supporting swapping backends good enough? If you can load the CUDA backend then can't you use that with n_gpu_layers = 0 instead of reloading to a pure CPU backend?

[saddam213]

Just support for swapping would be handy,

I am one of the few that run LLamaSharp on servers with GPUs but don't want to use GPU's as they are busy running other apps, so Just hoping we include a way to override the incoming auto-dectect code

In the past n_gpu_layers = 0 resulted in a crash, this could have been resolved as its been several months since I tried

[AsakusaRinne]

To me either supporting multiple backends or swapping backends seems too aggressive now. The largest problem is that there're memories held in native library which we know few about. Things may be out of our control when swapping the backend. I suggest to allow user to select a preferred backend only once before loading llama model now (the "ability to configure the backend before it is loaded" discussed above).

In the condition @saddam213 mentioned, there's a point which needs to be further discussed. If there's a model m running on GPU and now I want it to run only on cpu instead, is the time of re-loading model and context acceptable? Besides, when GPUs are busy running other apps, is the gpu memory also in short? The answer decides whether it's supposed to clear the memories allocated in GPU.
If clearing it every time is ok, I think save the current states and reloading the whole process with another backend is reasonable. Otherwise it will be complex. If we switch to another backend, there's a huge risk of memory leak. Even if we solved the memory leak, after executing some inference with the swapped backend, how can we synchronize the new state with the old one?

When we call NativeLibrary.Free, the memories allocated by dll on heap won't be released. If we want to avoid memory leak, we must have an API in c++ to clear all the memories allocated itself. Then we call the API from C# before loading another backend.

Though I'm against dynamically switching backend after loading now, I think this issue is a commonly encountered one when building a service base on LLamaSharp and worth working on. I was just saying we shouldn't make the changes until we eliminate the risks.

[hswlab]

I'm working on an Electron/LLamaSharp application with .NET6 and was wondering if it would be possible to give the user the option to select the backend in the APP settings via a dropdown. The dll would then be downloaded from Github (https://github.com/ggerganov/llama.cpp/releases/latest) and unpacked. The application would then need to be restarted. When starting the application, LLamaSharp just needs to know in which directory the unpacked dll is located.

I noticed this code, where an alternative path to the backend dll could possibly be loaded.

I havn't tested the NativeLibraryConfig yet, but this looks very promising for dynamic loading of the backend.
Would it work to call the NativeLibraryConfig.WithLibrary method in the Program.cs at application start with a custom path?

Edit:
It seems that WithLibrary(string libraryPath) is not a static method. I think I can't use it so easily as I thought :)

Edit2:
I think I figured it out how to use this config. I set this at the beginning of the Program.cs:
NativeLibraryConfig.Default.WithLibrary("C:\Users\my_path\llama.dll");

Now the correct path is loaded, but the exception is still called. Probably I'm loading the wrong dll or I'm missing something. I will check it tomorrow ^^'.

[AsakusaRinne]

Edit: It seems that WithLibrary(string libraryPath) is not a static method. I think I can't use it so easily as I thought :)

Please use NativeLibraryConfig.Default.WithLibrary. Note that this API may be changed to NativeLibraryConfig.Instance in #281.

Would it work to call the NativeLibraryConfig.WithLibrary method in the Program.cs at application start with a custom path?

I think so, it was designed to do this :)

I haven't tested the NativeLibraryConfig yet, but this looks very promising for dynamic loading of the backend.

If you would like to, you can remove the check here to load the library anywhere at runtime to help to check if there's memory leak or other bad behaviours.

[martindevans]

Looks like you've already worked it out, but NativeLibraryConfig.Default.WithLibrary( is the way to do this :)

Please note though that you cannot just download the latest DLL from llama.cpp - you must download exactly right commit version. There is absolutely no compatibility from version to version!