[Autotuner] Use cudagraph for time measurement on Nvidia hardware

[yf225]

This should make timing on Nvidia hardware more accurate.

[njriasan]

LGTM! Thanks!

[Chillee]

doesn't this make autotuning time much slower?

[yf225]

doesn't this make autotuning time much slower?

yeah I think with original n_retries = 10 in the PR, the autotuning time was not good. I've just updated the PR to effectively set n_retries = 1 (since we are already doing cuda_graph_total_time / n_repeat to get average per-iteration runtime), and it now brings the autotuning time on par with the original time.

[jansel]

How are you testing that this is giving more stable measurements?

Can you share some data?

[jansel]

Does the above also use median? Median tends to remove outliers.

[jansel]

Why?

[jansel]

Do the numbers from this match those from the interleaved_bench we are using for more precise measurements? If not it could cause some issues with the rebench logic.

This benchmarking function isn't the main one we use, we use it to get a rough initial time then do interleaved benchmarking on the best configs.

[yf225]

This benchmarking function isn't the main one we use, we use it to get a rough initial time then do interleaved benchmarking on the best configs.

ah I see, then I think the current interleaved_bench approach is still more robust. The most precise approach would be interleaved + cudagraph, but cudagraph requires running a few iterations within the same graph (N kernels * K iters/kernel, and each timing region (graph) contains K iterations) to amortize the overhead, while currently interleaved_bench does per-iteration CUDA event timing (N kernels * K iters/kernel, each timing region only contains 1 iteration).

We could split the K further into K1 * K2, and cudagraph over K2 iterations as one timing region. But I am slightly unsure if this is worth doing given the higher complexity.