Skip to content

Commit a65e012

Browse files
authored
docs(Vulkan): Add GPU docker documentation for Vulkan (#5255)
Add GPU docker documentation for Vulkan
1 parent 8e9b41d commit a65e012

File tree

1 file changed

+33
-0
lines changed

1 file changed

+33
-0
lines changed

docs/content/docs/features/GPU-acceleration.md

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -278,3 +278,36 @@ docker run --rm -ti --device /dev/dri -p 8080:8080 -e DEBUG=true -e MODELS_PATH=
278278
```
279279

280280
Note also that sycl does have a known issue to hang with `mmap: true`. You have to disable it in the model configuration if explicitly enabled.
281+
282+
## Vulkan acceleration
283+
284+
### Requirements
285+
286+
If using nvidia, follow the steps in the [CUDA](#cudanvidia-acceleration) section to configure your docker runtime to allow access to the GPU.
287+
288+
### Container images
289+
290+
To use Vulkan, use the images with the `vulkan` tag, for example `{{< version >}}-vulkan-ffmpeg-core`.
291+
292+
#### Example
293+
294+
To run LocalAI with Docker and Vulkan, you can use the following command as an example:
295+
296+
```bash
297+
docker run -p 8080:8080 -e DEBUG=true -v $PWD/models:/build/models localai/localai:latest-vulkan-ffmpeg-core
298+
```
299+
300+
### Notes
301+
302+
In addition to the commands to run LocalAI normally, you need to specify additonal flags to pass the GPU hardware to the container.
303+
304+
These flags are the same as the sections above, depending on the hardware, for [nvidia](#cudanvidia-acceleration), [AMD](#rocmamd-acceleration) or [Intel](#intel-acceleration-sycl).
305+
306+
If you have mixed hardware, you can pass flags for multiple GPUs, for example:
307+
308+
```bash
309+
docker run -p 8080:8080 -e DEBUG=true -v $PWD/models:/build/models \
310+
--gpus=all \ # nvidia passthrough
311+
--device /dev/dri --device /dev/kfd \ # AMD/Intel passthrough
312+
localai/localai:latest-vulkan-ffmpeg-core
313+
```

0 commit comments

Comments
 (0)