How to use vulkan to call the GPU of an Android device（without use Opencl？）

[qtyandhasee]

I want to cross-compile Android on x86_64 linux
want to use vulkan to call Gpus on Android devices.
Does llama.cpp provide the corresponding documentation for cross-compilation?

Compilation platform: x86_64 linux（RTX4090)
Target platform:Android（Adreno740v2)

compile
export ANDROID_NDK=path/to/android-ndk-r26c
mkdir build_cpu_android && cd build_cpu_android
cmake .. -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake \ -DCMAKE_BUILD_TYPE=Release \ -DANDROID_ABI="arm64-v8a"\ -DANDROID_STL=c++_static \ -DANDROID_NATIVE_API_LEVEL=android-21 \ -DLLAMA_VULKAN=1
make -j8

model use
llama-2-7b-chat.Q4_K_M.gguf

vulkan installation
I downloaded the vulkan 1.2 sdk package from the official website, extracted it and stored it in the./vulkan directory, and edited the bashrc file to set environment variables

nano ~/.bashrc export VULKAN_SDK=/root/nfs/llama/Galaxy-llama.cpp/vulkan/1.2.198.1/x86_64 && export PATH=$VULKAN_SDK/bin:$PATH && export LD_LIBRARY_PATH=$VULKAN_SDK/lib:$LD_LIBRARY_PATH && export LIBRARY_PATH=$VULKAN_SDK/lib:$LIBRARY_PATH && export VK_LAYER_PATH=$VULKAN_SDK/etc/vulkan/explicit_layer.d

The problems I have encountered so far
the build info:vulkan cannot be found (I have confirmed that the vulkan environment variable was successfully imported)

when use make,
error: use of undeclared identifier 'POSIX_MADV_WILLNEED'
if (posix_madvise(addr, std::min(file->size, prefetch), POSIX_MADV_WILLNEED)) {
^
/CodeSpace/nfs/llama/Galaxy-llama.cpp/llama.cpp:1144:49: error: use of undeclared identifier 'POSIX_MADV_RANDOM'
if (posix_madvise(addr, file->size, POSIX_MADV_RANDOM)) {
^

[4onen]

I'm unable to directly help with your use case, but I was able to successfully build llama.cpp with Vulkan support in the Termux terminal emulator app on my Pixel 8 (Arm-v8a CPU, Mali G715 GPU) with the OpenCL packages not installed. I was also able to build llama.cpp with OpenCL support in the same way with the Vulkan packages unisntalled. (In both cases, offloading layers resulted in a significant performance decrease due to unified RAM contention.)

I'm not clear what factor OpenCL plays in the problem you're describing here (Specifically, failing to get the Vulkan headers necessary to build.)

[qtyandhasee]

I'm unable to directly help with your use case, but I was able to successfully build llama.cpp with Vulkan support in the Termux terminal emulator app on my Pixel 8 (Arm-v8a CPU, Mali G715 GPU) with the OpenCL packages not installed. I was also able to build llama.cpp with OpenCL support in the same way with the Vulkan packages unisntalled. (In both cases, offloading layers resulted in a significant performance decrease due to unified RAM contention.)

I'm not clear what factor OpenCL plays in the problem you're describing here (Specifically, failing to get the Vulkan headers necessary to build.)

@4onen Thanks for your answer! At least tell me it's possible to succeed on Android using llama.cpp vulkan. I wonder how you compile it? I'm confused about this now because the latest readme says that VULKAN backend support is broken and I'm not using Termux as suggested by the readme. I compiled on jetson XN and scp transferred the results to the corresponding directory on a real Android device. I'm struggling with vulkan's bug right now...

[Jeximo]

At least tell me it's possible to succeed on Android using llama.cpp vulkan. I wonder how you compile it?

It's possible to compile Vulkan on Android in Termux. This includes installing packages such as vulkan-headers, and probably a vulkan-loader from Termux.

Running Vulkan llama.cpp in Termux on Android isn't currently possible as far as I know. I don't recall the error message last time I tried.

Related: Vulkan for Android #5186

[4onen]

Ah, my phone has a Mali GPU, which explains why it was able to work for me but not in that issue. I'll bow out now to let y'all focus on the specific hardware facing this problem.

[woachk]

Running Vulkan llama.cpp in Termux on Android isn't currently possible as far as I know.

It is. But was only able to get passable perf on the Kompute backend when using Adreno (w/ hacks). The more regular Vulkan backend blows up quite quickly, and even when that's worked around, I get garbled output at best

[akingoverlook]

I want to cross-compile Android on x86_64 linux want to use vulkan to call Gpus on Android devices. Does llama.cpp provide the corresponding documentation for cross-compilation?

Compilation platform: x86_64 linux（RTX4090) Target platform:Android（Adreno740v2)

compile export ANDROID_NDK=path/to/android-ndk-r26c mkdir build_cpu_android && cd build_cpu_android cmake .. -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake \ -DCMAKE_BUILD_TYPE=Release \ -DANDROID_ABI="arm64-v8a"\ -DANDROID_STL=c++_static \ -DANDROID_NATIVE_API_LEVEL=android-21 \ -DLLAMA_VULKAN=1 make -j8

model use llama-2-7b-chat.Q4_K_M.gguf

vulkan installation I downloaded the vulkan 1.2 sdk package from the official website, extracted it and stored it in the./vulkan directory, and edited the bashrc file to set environment variables

nano ~/.bashrc export VULKAN_SDK=/root/nfs/llama/Galaxy-llama.cpp/vulkan/1.2.198.1/x86_64 && export PATH=$VULKAN_SDK/bin:$PATH && export LD_LIBRARY_PATH=$VULKAN_SDK/lib:$LD_LIBRARY_PATH && export LIBRARY_PATH=$VULKAN_SDK/lib:$LIBRARY_PATH && export VK_LAYER_PATH=$VULKAN_SDK/etc/vulkan/explicit_layer.d

The problems I have encountered so far the build info:vulkan cannot be found (I have confirmed that the vulkan environment variable was successfully imported) when use make, error: use of undeclared identifier 'POSIX_MADV_WILLNEED' if (posix_madvise(addr, std::min(file->size, prefetch), POSIX_MADV_WILLNEED)) { ^ /CodeSpace/nfs/llama/Galaxy-llama.cpp/llama.cpp:1144:49: error: use of undeclared identifier 'POSIX_MADV_RANDOM' if (posix_madvise(addr, file->size, POSIX_MADV_RANDOM)) { ^

Your Vulkan not found error is due to cmake not knowing how to find Vulkan for the cross-compiler. You need to tell it where your Vulkan SDK is, on the command line.

The other errors from posix calls are due to using a very old Android API level (21). You need to bump that to something current, like 33.

cmake -DLLAMA_VULKAN=1
-DVulkan_INCLUDE_DIR=~/Vulkan-Headers/include
-DCMAKE_TOOLCHAIN_FILE=$NDK/build/cmake/android.toolchain.cmake
-DANDROID_ABI=arm64-v8a
-DANDROID_PLATFORM=android-33
-DCMAKE_C_FLAGS=-march=armv8.4a+dotprod+i8mm ..

But you will be mostly wasting your time with this endeavor. The Vulkan backend works OK on "typical" GPUs, but for Adreno it needs some work to even load, and then not die immediately, and then a lot more work to get it to perform adequately.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[Utkarsh4430]

For reference, here's the command I used to cross-compile Vulkan for a Samsung mobile device:

cmake \
-DCMAKE_TOOLCHAIN_FILE=$NDK/build/cmake/android.toolchain.cmake \
-DANDROID_ABI=arm64-v8a \
-DANDROID_PLATFORM=android-23 \
-DCMAKE_C_FLAGS=-march=armv8.4a+dotprod .. \
-DLLAMA_VULKAN=1 \
-DVulkan_INCLUDE_DIR={path to the "include" directory after building Vulkan-Headers repo}  \
-DVulkan_LIBRARY={path to libvulkan.so compiled for aarch64} \
-DLLAMA_OPENMP=OFF

Thanks!

[andrewson97]

-- The C compiler identification is Clang 18.0.3
-- The CXX compiler identification is Clang 18.0.3
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/iterate/Andrew/android-ndk-r27c/toolchains/llvm/prebuilt/linux-x86_64/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/iterate/Andrew/android-ndk-r27c/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: /usr/bin/git (found version "2.34.1")
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
-- Found OpenMP_C: -fopenmp=libomp
-- Found OpenMP_CXX: -fopenmp=libomp
-- Found OpenMP: TRUE
-- OpenMP found
CMake Warning at ggml/src/CMakeLists.txt:274 (message):
AMX requires gcc version > 11.0. Turning off GGML_AMX.

-- ccache found, compilation results will be cached. Disable with GGML_CCACHE=OFF.
-- CMAKE_SYSTEM_PROCESSOR: aarch64
-- ARM detected
-- Performing Test COMPILER_SUPPORTS_FP16_FORMAT_I3E
-- Performing Test COMPILER_SUPPORTS_FP16_FORMAT_I3E - Failed
-- Configuring done
-- Generating done
CMake Warning:
Manually-specified variables were not used by the project:

LLAMA_OPENMP
LLAMA_VULKAN
Vulkan_INCLUDE_DIR
Vulkan_LIBRARY

Why it is not using the vulkan paths?

[harryaskham]

I recently got this working in Nix-on-Droid (a Nix-based Termux fork) on a Samsung Galaxy S24 Ultra, in case useful to anyone that finds this.

I posted some very preliminary benchmarks at nix-community/nix-on-droid#305 (comment). Initial testing shows a ~2x speedup with Vulkan, but only under certain models; with too long a prompt, or with Qwen models, we sometimes get garbage output (random UTF8 or just "GGGGG...") on the GPU, even with -fa. I'm skeptical about the 2x and wonder if the CPU version could be optimised to make these equal, since they use the same shared memory and the Adreno 750 GPU is missing most ops that'd provide this boost.

I have a llama.cpp fork that just adds a (shell.nix)[https://github.com/harryaskham/llama.cpp-vulkan-aarch64/blob/master/shell.nix] in which one can:

> nix-shell                                                                                                                                                                                                                                                                                                                                                                                                          
$ build-llama-cpp                                                                                                                                                                                                                                                                                                                                                                                                    
$ ./build/bin/llama-cli -m ...   

I didn't need many of the other env vars mentioned in this thread but there are many confounding factors; I'll try some and see if I can boost performance / get around the corruption issues.