"Illegal Instruction" on model load on Android-Termux for v1.54+

[Mizstik]

In v1.54 and in latest, although it compiles successfully (no error), koboldcpp fails at the step when it's about to load the model (after having shown System Info) with "Illegal Instruction" and then exit to shell.

I did a git checkout v1.52, make clean, then make. And with this, it still runs fine. I didn't try 1.53, sorry.

Something happened after 1.52 that broke Android compatibility (aarch64) at least in Termux, and possibly other ARM-based SBCs. My particular phone is ROG Phone 6. (SD8 Gen 1, 16 GB RAM)

[LostRuins]

Sorry, can I check which model you tried this with?

[Mizstik]

A bunch of models I happened to have on the phone at the time. If I remember correctly:

mistral-7b-instruct-v0.2.Q3_K_L.gguf
guanaco-3b-uncensored-v2.ggmlv1.q4_0.bin
orca_mini_v3_7b.ggmlv3.q3_K_M.bin

They all worked after I reverted to v1.52.

[Dravoss]

I have the very same problem, i am running it on an android with Aarch64 on termux and the last version that is working in my phone is 1.53, i have tried a bunch of models and to download the .zip manually and building it with make but i always got the same log when i run it:

Welcome to KoboldCpp - Version 1.54
Warning: OpenBLAS library file not found. Non-BLAS library will be used.
Initializing dynamic library: koboldcpp_default.so

Namespace(model=None, model_param='model1-7b.Q5_K_M.gguf', port=5001, port_param=5001, host='', launch=False, lora=None, config=None, threads=3, blasthreads=3, highpriority=False, contextsize=2048, blasbatchsize=512, ropeconfig=[0.0, 10000.0], smartcontext=False, noshift=False, bantokens=None, forceversion=0, nommap=False, usemlock=False, noavx2=False, debugmode=0, skiplauncher=False, hordeconfig=None, noblas=False, useclblast=None, usecublas=None, gpulayers=0, tensor_split=None, onready='', multiuser=0, remotetunnel=False, foreground=False, preloadstory='', quiet=False, ssl=None)

Loading model: /data/data/com.termux/files/home/koboldcpp-1.54/model1-7b.Q5_K_M.gguf
[Threads: 3, BlasThreads: 3, SmartContext: False, ContextShift: True]

The reported GGUF Arch is: llama

Identified as LLAMA model: (ver 6)
Attempting to Load...

Using automatic RoPE scaling. If the model has customized RoPE settings, they will be used directly instead!
System Info: AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 |
Illegal _instruction

[gustrd]

I wonder if it's a problem with a Termux update. Can you still build the older version?

[Mizstik]

I wasn't updating Termux (or any package) at all when I discovered this issue. It's typical to freeze Termux updates because we don't want the F-Droid version to be overwritten by the Play Store version. I haven't run pkg update in forever either and only git pulled koboldcpp. (Though after I discovered the issue, I went around to update everything to see if any updated packages would fix the issue but no dice.)

So after all that, I tried downgrading to v1.52 and that works. The other guy above said that v1.53 also works. v1.54 and above don't work.

Also note that they all built fine. They fail at runtime, during model load.

[Dravoss]

I wonder if it's a problem with a Termux update. Can you still build the older version?

Yes 1.53 still builds and run just fine

[LostRuins]

I will take a look

[Dravoss]

I will take a look

Tell me if you need me to test anything.

[LostRuins]

Hi everyone, I just did a clean install on the latest experimental for guanaco and phi, both seem to be working.
How did you setup the install? Can you try these steps:

• Install and run Termux from F-Droid
• Enter the command termux-change-repo and choose Mirror by BFSU
• Install dependencies with pkg install wget git python (plus any other missing packages)
• Install dependencies apt install openssl (if needed)
• Clone the experimental branch of the repo git clone -b concedo_experimental https://github.com/LostRuins/koboldcpp.git
• Navigate to the koboldcpp folder cd koboldcpp
• Build the project make
• Grab a small GGUF model, such as wget https://huggingface.co/TheBloke/phi-2-GGUF/resolve/main/phi-2.Q2_K.gguf
• Start the python server python koboldcpp.py --model phi-2.Q2_K.gguf
• Connect to http://localhost:5001 on your mobile browser

Tell me:

• If it works, good
• If it fails, which part didn't work, and what error do you get?

I'm wondering if someone somewhere pushed a bad package for something. Would be good to describe how you originally setup vs the above steps if there's any differences.

[Dravoss]

Unfortunately i am getting the same error.

For my original setups i tried to clone the hub and manually downloading the zip from releases but both failed to load the model.

This is the log after following your last post:

~/koboldcpp $ python koboldcpp.py --model phi-2.Q2_K.gguf

---

Welcome to KoboldCpp - Version 1.56
Warning: OpenBLAS library file not found. Non-BLAS library will be used.
Initializing dynamic library: koboldcpp_default.so

Namespace(model='phi-2.Q2_K.gguf', model_param='phi-2.Q2_K.gguf', port=5001, port_param=5001, host='', launch=False, lora=None, config=None, threads=3, blasthreads=3, highpriority=False, contextsize=2048, blasbatchsize=512, ropeconfig=[0.0, 10000.0], smartcontext=False, noshift=False, bantokens=None, forceversion=0, nommap=False, usemlock=False, noavx2=False, debugmode=0, skiplauncher=False, hordeconfig=None, noblas=False, useclblast=None, usecublas=None, gpulayers=0, tensor_split=None, onready='', multiuser=0, remotetunnel=False, foreground=False, preloadstory='', quiet=False, ssl=None)

Loading model: /data/data/com.termux/files/home/koboldcpp/phi-2.Q2_K.gguf
[Threads: 3, BlasThreads: 3, SmartContext: False, ContextShift: True]

The reported GGUF Arch is: phi2

Identified as LLAMA model: (ver 6)
Attempting to Load...

Using automatic RoPE scaling. If the model has customized RoPE settings, they will be used directly instead!
System Info: AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 |
Illegal instruction
~/koboldcpp $

[Mizstik]

Still no go. Here's my Termux log from start to finish. I uninstalled Termux and then reinstalled again fresh from F-Droid. The log starts from changing the repo and including the compile outputs, wget, and the koboldcpp execution up until the last line which is Illegal instruction.

termux_transcript.txt

[Mizstik]

@LostRuins Which phone are you using? If I have the same or a similar phone, I can go try.

[LostRuins]

I was testing on a Samsung Galaxy S9 Plus

[Mizstik]

Hmm I don't have that one but I do have a phone also with SD845 (Poco F1). I'll give it a try.

[Dravoss]

Mine is a rog phone 6 with aarch64:

Android 12
Qualcomm® Snapdragon® 8+ Gen 1 Mobile Platform
Qualcomm® Adreno™ 730
LPDDR5 16GB