CUDA an illegal memory access was encountered

[piotr-sikora-v]

When I try to start any large model I have this error:

CUDA error 700 at ggml-cuda.cu:8303: an illegal memory access was encountered

my GPU is 1080 Ti (11GB vRAM).
model base.en works.

openai-whisper with large-v2 works without problem using GPU.

I also tried to run quantize model... q5_0 have same problem... q4_0 start but I don't have any output :/

on dmesg:

[12337.297750] NVRM: Xid (PCI:0000:04:00): 31, pid=167607, name=main, Ch 00000008, intr 10000000. MMU Fault: ENGINE GRAPHICS GPCCLIENT_T1_8 faulted @ 0x7ffb_7fe03000. Fault is of type FAULT_PDE ACCESS_TYPE_READ

example run:

# ./main -debug -m models/ggml-large-v3.bin -l pl   janina.wav 
whisper_init_from_file_with_params_no_state: loading model from 'models/ggml-large-v3.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51866
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 1280
whisper_model_load: n_audio_head  = 20
whisper_model_load: n_audio_layer = 32
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 1280
whisper_model_load: n_text_head   = 20
whisper_model_load: n_text_layer  = 32
whisper_model_load: n_mels        = 128
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 5 (large v3)
whisper_model_load: adding 1609 extra tokens
whisper_model_load: n_langs       = 100
ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
  Device 0: NVIDIA GeForce GTX 1080 Ti, compute capability 6.1
whisper_backend_init: using CUDA backend
whisper_model_load:     CUDA buffer size =  3117.87 MB
whisper_model_load: model size    = 3117.39 MB
whisper_backend_init: using CUDA backend
whisper_init_state: kv self size  =  220.20 MB
whisper_init_state: kv cross size =  245.76 MB
whisper_init_state: compute buffer (conv)   =   32.36 MB
whisper_init_state: compute buffer (encode) =  212.36 MB
whisper_init_state: compute buffer (cross)  =    9.32 MB
whisper_init_state: compute buffer (decode) =   99.17 MB

system_info: n_threads = 4 / 30 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | METAL = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | CUDA = 1 | COREML = 0 | OPENVINO = 0 | 

main: processing 'janina.wav' (50387302 samples, 3149.2 sec), 4 threads, 1 processors, 5 beams + best of 5, lang = pl, task = transcribe, timestamps = 1 ...


CUDA error 700 at ggml-cuda.cu:8303: an illegal memory access was encountered
current device: 0

[kornpow]

try using one of the smaller models.

I have a similar problem, but I get sort of garbage output when using GPU, when it works flawlessly with the CPU nvm that problem got solved when i installed cuda properly

btw I am running a GTX 1080

[ggerganov]

Are you using the latest version of this repo?

[piotr-sikora-v]

Are you using the latest version of this repo?

yes, I update repo few minutes ago, do make clean, new build... same issue :/

[SmallAndSoft]

I am having exactly the same issue with GTX 1060 while using ggml-large-v3-q5_0.bin

[ggerganov]

Would need a sample audio and the exact command that reproduces the issue.

[kornpow]

./main -m models/ggml-large-v3.bin -f samples/jfk.wav

whisper_init_from_file_with_params_no_state: loading model from 'models/ggml-large-v3.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51866
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 1280
whisper_model_load: n_audio_head  = 20
whisper_model_load: n_audio_layer = 32
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 1280
whisper_model_load: n_text_head   = 20
whisper_model_load: n_text_layer  = 32
whisper_model_load: n_mels        = 128
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 5 (large v3)
whisper_model_load: adding 1609 extra tokens
whisper_model_load: n_langs       = 100
ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
  Device 0: NVIDIA GeForce GTX 1080, compute capability 6.1
whisper_backend_init: using CUDA backend
whisper_model_load:     CUDA buffer size =  3117.87 MB
whisper_model_load: model size    = 3117.39 MB
whisper_backend_init: using CUDA backend
whisper_init_state: kv self size  =  220.20 MB
whisper_init_state: kv cross size =  245.76 MB
whisper_init_state: compute buffer (conv)   =   32.36 MB
whisper_init_state: compute buffer (encode) =  212.36 MB
whisper_init_state: compute buffer (cross)  =    9.32 MB
whisper_init_state: compute buffer (decode) =   99.17 MB

system_info: n_threads = 4 / 8 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | METAL = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | CUDA = 1 | COREML = 0 | OPENVINO = 0 | 

main: processing 'samples/jfk.wav' (176000 samples, 11.0 sec), 4 threads, 1 processors, 5 beams + best of 5, lang = en, task = transcribe, timestamps = 1 ...


CUDA error 700 at ggml-cuda.cu:8303: an illegal memory access was encountered
current device: 0

Wed Nov 22 10:31:40 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1080        Off | 00000000:01:00.0 Off |                  N/A |
|  0%   34C    P0              51W / 210W |     15MiB /  8192MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

[ggerganov]

If you apply the following patch, does it work?

diff --git a/ggml-cuda.cu b/ggml-cuda.cu
index b420330..9da239a 100644
--- a/ggml-cuda.cu
+++ b/ggml-cuda.cu
@@ -96,7 +96,7 @@
 // -  7B quantum model: +100-200 MB
 // - 13B quantum model: +200-400 MB
 //
-//#define GGML_CUDA_FORCE_MMQ
+#define GGML_CUDA_FORCE_MMQ
 
 // TODO: improve this to be correct for more hardware
 //       for example, currently fails for GeForce GTX 1660 which is TURING arch (> VOLTA) but does not have tensor cores

[Zubairu]

I'm using ggml-large-v3.bin too, and also have this issue with Quadro P6000.

after apply the patch above, still no lucky.

[ggerganov]

Yeah, I'm sorry. Without being able to reproduce, I won't be able to fix it. It works on my old GTX 1660 and I can't rent an older GPU to test

[ggerganov]

Try #1548, but unlikely that it will resolve the problem

[Zubairu]

Try #1548, but unlikely that it will resolve the problem

thanks for you reply, i tried #1548 , but still have the same issue for large-v3, while large-v2 works fine.

[ggerganov]

but still have the same issue for large-v3, while large-v2 works fine.

Hm, interesting. Could the rest of the people that have issues confirm that it is only v3 that does not work?
@SmallAndSoft @piotr-sikora-v @kornpow

[cebtenzzre]

I am hitting this with large-v1 in f16 and q5_1:

CUDA error 700 at ggml-cuda.cu:8335: an illegal memory access was encountered

I do not hit this with q8_0 but I get gibberish. This model works fine on 4774d2f (with -arch=native for modern CUDA). Will bisect.

edit:
ec7a6f0 works fine on my P40.
b050283 (#1472) either hangs (apparently doing nothing with 100% GPU usage), or crashes with the above assertion failure on line 8224.

[ggerganov]

Before b050283 there was just naive matrix multiplication with host-device-host copy running on the GPU. This was the first commit to introduce llama-like GPU offloading of the entire graphs.

But it seems that we have some bug in ggml-cuda.cu and it looks like it triggers on older hardware. This is a bit worrying