Issue #350 development

[Onkitova]

Related to: #350

Nvidia CUDA binaries are taken from archives:

• CUDA 11 (cudart-llama-bin-win-cu11.7.1-x64.zip)
• CUDA 12 (cudart-llama-bin-win-cu12.2.0-x64.zip) from the latest (at the moment of writing this) build of ggerganov's llama.cpp.

Editing .nuspec at this point is discussible, however.

[Onkitova]

@martindevans @AsakusaRinne
can confirm everything works locally as expected (no cuda toolkit installed, yet cuda is utilized by LlamaSharp/Examples project), but unfortunately I wasn't able to push nvidia binaries itself (due to its size, I guess). Please help.
Originally, commit looked like that:

[SanftMonster]

I don't think we should include the files in the repo because the sum of the sizes could reach more than 500M. Though it's best to automatically include them in ci to upload to release, I don't mind uploading them manually.

The main concern at my side is how users include them in their projects. Here's a way I'd like to suggest:

1. Add an API to NativeLibraryConfig, such as WithCublasDependency(string folder) to allow users specify a path of these dependencies.
2. In NativeApi.Load.cs, after having got which native library to load, copy the dependencies specified by user to the same path of the selected native library.

Therefore users will have three ways to use the dependencies:

1. Add the dependency folder to the PATH environment variables.
2. Manually copy the dependencies to the same folder of used native library.
3. Specify the path of dependencies in the code.

[SanftMonster]

As discussed before, we'll add these files in the release instead of nuget package. Though I won't be against to distributing a new backend package with all these files, I don't think it's a good idea to add them into existed package. :)

[Onkitova]

@AsakusaRinne
Okay, I am totally agree with your vision, but this whole pull request is more like a proof of concept, answering the last comment from @martindevans in #350.
Like I said, I have no experience with ci/cd githubs stuff including related scripting and, unfortunately, I don't have enough time right now to get into it all seriously.
I was asked how it could look like (working version) -- I showed you. Further details of the final implementation, which is supposed to become a part of release version, are beyond my competence at the moment, sorry.
Once again, for the sake of convenience, I am totally okay with you guys rejecting or editing this PL, since it was just a form of answer, continuing our discussion in #350.

[SanftMonster]

@Onkitova I see. We'll handle the part of CI/CD and thanks for all your works and suggestions about this issue. :)

[martindevans]

Thanks for investigating this @Onkitova, it definitely confirms the theory with the CUDA redist even if those massive files sizes are a bit of a pain!

[martindevans]

@AsakusaRinne How about publishing 2 new nuget packages with the cudart binaries:

cuda11.7.1.runtime

• cu11.7.1/cublas64_11(.dll/.so)
• cu11.7.1/cublasLt64_11(.dll/.so)
• cu11.7.1/cudart64_110(.dll/.so)

cuda12.1.0.runtime

• cu12.1.0/cublas64_12(.dll/.so)
• cu12.1.0/cublasLt64_12(.dll/.so)
• cu12.1.0/cudart64_12(.dll/.so)

Then we can depend on those packages in our 2 CUDA backends. Since those dependencies will never change we won't have to worry about handling huge files every time we push out an update.

[SanftMonster]

@AsakusaRinne How about publishing 2 new nuget packages with the cudart binaries:

cuda11.7.1.runtime

• cu11.7.1/cublas64_11(.dll/.so)
• cu11.7.1/cublasLt64_11(.dll/.so)
• cu11.7.1/cudart64_110(.dll/.so)

cuda12.1.0.runtime

• cu12.1.0/cublas64_12(.dll/.so)
• cu12.1.0/cublasLt64_12(.dll/.so)
• cu12.1.0/cudart64_12(.dll/.so)

Then we can depend on those packages in our 2 CUDA backends. Since those dependencies will never change we won't have to worry about handling huge files every time we push out an update.

It's definitely a good idea!

[martindevans]

I'll close this PR, since we've decided on a different course of action (separate cudart nuget packages). We can track that back over in #350