GEMM kernel regression slipping past llama-perplexity?

[tlemo]

While experimenting with GEMM kernels (CUDA), I introduced a bug. The perplexing part is that all the unit tests (test-backend-ops) were passing, and the PPL score was exactly the same as before the change which introduced the bug, and the only way I noticed that something is wrong was playing with llama-cli.

The unit tests gap is understandable, since the coverage is not 100%, and the only case that was affected was an fp16 MUL_MAT with non-contiguous tensors of "irregular" dimensions. I assumed that unit tests + PPL score would catch regressions with high confidence, but it doesn't appear to be the case.

What would be a better way to catch functional regressions?

[am17an]

Without knowing what you changed, as I recently found out PPL tests are impervious to batch_size = 1 bugs.

[tlemo]

Thanks @am17an, I'll keep the batch_size == 1 in mind. Here, both llama-cli and llama-perplexity have batch_size > 1.