Bug: Inconsistent ggml-4-x86-cuda-v100 ci failures on master

[mofosyne]

Note: Only one datapoint of ci failure, but it would be important to keep track of this behavior over the next few commits

What happened?

• Passing commit

  • 504f0c3

• Failing commit, but its a readme only change

  • 0e8d8bf

Noticed that it said it's failing in 20 - test-backend-ops, it be good to identify the cause of this issue and potential ways to fix it. The failure in test #20 in test-backend-ops looked like below which doesn't seem to explain much to me. But hopefully it makes sense to someone else here.

[CPY] NMSE = 0.000003149 > 0.000000100 looks interesting however

Name and Version

between commit 504f0c3 and 0e8d8bf

What operating system are you seeing the problem on?

Other? (Please let us know in description)

Relevant log output

�[1;32mOK�[0m
  CPY(type_src=f32,type_dst=q4_1,ne=[256,4,4,4]): ggml_backend_cuda_graph_compute: disabling CUDA graphs due to GPU architecture
ggml_backend_cuda_graph_compute: disabling CUDA graphs due to GPU architecture
[CPY] NMSE = 0.000003149 > 0.000000100 ggml_backend_cuda_graph_compute: disabling CUDA graphs due to GPU architecture
ggml_backend_cuda_graph_compute: disabling CUDA graphs due to GPU architecture
ggml_backend_cuda_graph_compute: disabling CUDA graphs due to GPU architecture
�[1;31mFAIL�[0m
  CPY(type_src=f32,type_dst=q5_0,ne=[256,4,4,4]): ggml_backend_cuda_graph_compute: disabling CUDA graphs due to GPU architecture
ggml_backend_cuda_graph_compute: disabling CUDA graphs due to GPU architecture
ggml_backend_cuda_graph_compute: disabling CUDA graphs due to GPU architecture
ggml_backend_cuda_graph_compute: disabling CUDA graphs due to GPU architecture
ggml_backend_cuda_graph_compute: disabling CUDA graphs due to GPU architecture
�[1;32mOK�[0m

[ggerganov]

See this comment for explanation: #7425 (comment)

[mofosyne]

Is there a way to reasonably suppress, change the rounding approach, increase threshold or adapt to this error (e.g. only trigger an error if two or more trips in a test?)

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.