CTX Processing regression for Pascal - Commit 2b4ea35

[askmyteapot]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

There is a regression on Context processing introduced in commit 2b4ea35

This is specifically for Pascal (6.1) with 1/64th fp16 performance. Problem is worse with longer CTX, getting up to 6x slower by 8kCTX

  Device 0: Tesla P40, compute capability 6.1
| model                          |       size |     params | backend    | ngl |    threads | test       |              t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ---------: | ---------- | ---------------: |
| llama 13B mostly Q8_0          |  12.88 GiB |    13.02 B | CUDA       |  99 |          1 | pp 512     |    485.03 ± 0.34 |
| llama 13B mostly Q8_0          |  12.88 GiB |    13.02 B | CUDA       |  99 |          1 | tg 128     |     18.30 ± 0.00 |

build: daab3d7 (1421)

Current Behavior

ggml_init_cublas: found 1 CUDA devices:
  Device 0: Tesla P40, compute capability 6.1
| model                          |       size |     params | backend    | ngl |    threads | test       |              t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ---------: | ---------- | ---------------: |
| llama 13B mostly Q8_0          |  12.88 GiB |    13.02 B | CUDA       |  99 |          1 | pp 512     |    207.34 ± 0.28 |
| llama 13B mostly Q8_0          |  12.88 GiB |    13.02 B | CUDA       |  99 |          1 | tg 128     |     18.28 ± 0.01 |

build: 2b4ea35 (1422)

ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   yes
ggml_init_cublas: CUDA_USE_TENSOR_CORES: no
ggml_init_cublas: found 1 CUDA devices:
  Device 0: Tesla P40, compute capability 6.1
| model                          |       size |     params | backend    | ngl |    threads |   main_gpu | test       |              t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ---------: | ---------: | ---------- | ---------------: |
warning: cannot set main_device=1 because there are only 1 devices. Using device 0 instead.
| llama 13B mostly Q8_0          |  12.88 GiB |    13.02 B | CUDA       |  99 |          1 |          1 | pp 512     |    208.54 ± 0.58 |
| llama 13B mostly Q8_0          |  12.88 GiB |    13.02 B | CUDA       |  99 |          1 |          1 | tg 128     |     18.29 ± 0.00 |

build: 207b519 (1446)

Environment and Context

Please provide detailed information about your computer setup. This is important in case the issue is not reproducible except for under certain specific conditions.

• 5800X + 64GB DDR 3733

• 3060ti (8GB) + TESLA P40 (24GB)

• Operating System, e.g. for Linux: Windows 11

• SDK version, : MSVC 2022

$ python3 -- 3.10.11
$ Cmake --version 3.27.4

@LostRuins

[quarterturn]

Sounds like the same issue as mine (#3780)

[quarterturn]

Try a

git reset --hard b1421
git pull
make clean && make -j LLAMA_CUBLAS=1

I think after that commit is where the problem started

[quarterturn]

Switching to 'cuda-cublas-opts' branch fixed it for me.

[ggerganov]

@askmyteapot Can you check if branch try-fix-3869 fixes the issue with LLAMA_CUDA_FORCE_MMQ=1 set

[askmyteapot]

@ggerganov
Built with only cublas flag

ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
  Device 0: Tesla P40, compute capability 6.1
| model                          |       size |     params | backend    | ngl |    threads | test       |              t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ---------: | ---------- | ---------------: |
| llama 13B mostly Q8_0          |  12.88 GiB |    13.02 B | CUDA       |  99 |          1 | pp 512     |    209.00 ± 1.04 |
| llama 13B mostly Q8_0          |  12.88 GiB |    13.02 B | CUDA       |  99 |          1 | tg 128     |     13.81 ± 0.00 |

build: 22cc9be (1447)

Built with -DLLAMA_CUDA_FORCE_MMQ=ON and cublas on.

D:\llama.cpp\build\bin\Release>llama-bench.exe -m D:\text-generation-webui\models\MythoMax-L2-13b-gguf-q8_0.gguf -t 1
ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   yes
ggml_init_cublas: CUDA_USE_TENSOR_CORES: no
ggml_init_cublas: found 1 CUDA devices:
  Device 0: Tesla P40, compute capability 6.1
| model                          |       size |     params | backend    | ngl |    threads | test       |              t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ---------: | ---------- | ---------------: |
| llama 13B mostly Q8_0          |  12.88 GiB |    13.02 B | CUDA       |  99 |          1 | pp 512     |    485.27 ± 0.34 |
| llama 13B mostly Q8_0          |  12.88 GiB |    13.02 B | CUDA       |  99 |          1 | tg 128     |     18.32 ± 0.01 |

build: 22cc9be (1447)

Has corrected the issue.

But, is there any way the force MMQ compile time flag be a runtime flag?

[cebtenzzre]

I agree that it would be nice if we had a runtime flag that could enable quantized mat*mat, even on modern GPUs. It does use less VRAM.

[LostRuins]

You can, but unless you're willing to build multiple sets of CUDA kernels and swapping between them based on batch size, you may lose the small-batch MMQ optimizations since those are determined at compile time.

#if defined(CUDA_USE_TENSOR_CORES)
#define  MMQ_X_Q4_0_AMPERE 4
#define  MMQ_Y_Q4_0_AMPERE 32
#define NWARPS_Q4_0_AMPERE 4
#else
#define  MMQ_X_Q4_0_AMPERE 64
#define  MMQ_Y_Q4_0_AMPERE 128
#define NWARPS_Q4_0_AMPERE 4
#endif

or maybe some intermediate value between these 2 options could provide a good compromise?

I personally don't really benefit from the small batch optimization much, I am on RTX 2060 6GB which should be CC 7.5, but maybe my hardware is kinda crappy either way and cublas wasn't that special for me. So I am back to using MMQ with the original (large batch) values

[ggerganov]

@LostRuins I think you mentioned earlier that for full offload, the new version on RTX 2060 is faster compared to MMQ and that you observe a regression for not-fully offloaded models due to 1-2 GPU layers less.

How big is the latter regression? Is it a regression both for short and long (>1024) contexts?
If you can post some numbers for PP and TG would be helpful to get a sense of the impact of the change