Skip to content

Issue #350 development #371

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

Issue #350 development #371

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions LLama/LLamaSharp.Runtime.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,32 @@
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/osx-x64/libllama.dylib">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/osx-x64/native/libllama.dylib</Link>
</None>

<None Include="$(MSBuildThisFileDirectory)runtimes/deps/cu11.7.1/cublas64_11.dll">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>cublas64_11.dll</Link>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should include the files in the repo because the sum of the sizes could reach more than 500M. Though it's best to automatically include them in ci to upload to release, I don't mind uploading them manually.

The main concern at my side is how users include them in their projects. Here's a way I'd like to suggest:

  1. Add an API to NativeLibraryConfig, such as WithCublasDependency(string folder) to allow users specify a path of these dependencies.
  2. In NativeApi.Load.cs, after having got which native library to load, copy the dependencies specified by user to the same path of the selected native library.

Therefore users will have three ways to use the dependencies:

  1. Add the dependency folder to the PATH environment variables.
  2. Manually copy the dependencies to the same folder of used native library.
  3. Specify the path of dependencies in the code.

</None>
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/cu11.7.1/cublasLt64_11.dll">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>cublasLt64_11.dll</Link>
</None>
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/cu11.7.1/cudart64_110.dll">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>cudart64_110.dll</Link>
</None>

<None Include="$(MSBuildThisFileDirectory)runtimes/deps/cu12.1.0/cublas64_12.dll">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>cublas64_12.dll</Link>
</None>
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/cu12.1.0/cublasLt64_12.dll">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>cublasLt64_12.dll</Link>
</None>
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/cu12.1.0/cudart64_12.dll">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>cudart64_12.dll</Link>
</None>
</ItemGroup>
</Project>
3 changes: 3 additions & 0 deletions LLama/runtimes/build/LLamaSharp.Backend.Cuda11.nuspec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@

<file src="runtimes/deps/cu11.7.1/libllama.dll" target="runtimes\win-x64\native\cuda11\libllama.dll" />
<file src="runtimes/deps/cu11.7.1/libllama.so" target="runtimes\linux-x64\native\cuda11\libllama.so" />
<file src="runtimes\deps\cu11.7.1\cublas64_11.dll" target="cublas64_11.dll" />
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed before, we'll add these files in the release instead of nuget package. Though I won't be against to distributing a new backend package with all these files, I don't think it's a good idea to add them into existed package. :)

<file src="runtimes\deps\cu11.7.1\cublasLt64_11.dll" target="cublasLt64_11.dll" />
<file src="runtimes\deps\cu11.7.1\cudart64_110.dll" target="cudart64_110.dll" />

<file src="icon512.png" target="icon512.png" />
</files>
Expand Down
3 changes: 3 additions & 0 deletions LLama/runtimes/build/LLamaSharp.Backend.Cuda12.nuspec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@

<file src="runtimes/deps/cu12.1.0/libllama.dll" target="runtimes\win-x64\native\cuda12\libllama.dll" />
<file src="runtimes/deps/cu12.1.0/libllama.so" target="runtimes\linux-x64\native\cuda12\libllama.so" />
<file src="runtimes\deps\cu12.1.0\cublas64_12.dll" target="cublas64_12.dll" />
<file src="runtimes\deps\cu12.1.0\cublasLt64_12.dll" target="cublasLt64_12.dll" />
<file src="runtimes\deps\cu12.1.0\cudart64_12.dll" target="cudart64_12.dll" />

<file src="icon512.png" target="icon512.png" />
</files>
Expand Down