[BugFix] Make FlashInferMetadataBuilder non-blocking (#25040)

Signed-off-by: Julien Lin <[email protected]>
Co-authored-by: Michael Goin <[email protected]>
Signed-off-by: yewentao256 <[email protected]>

Author: 2 people

vllm/v1/attention/backends/flashinfer.py

@@ -585,9 +585,10 @@ def build(self,
                         kv_data_type=self.kv_cache_dtype,
                     )
                 else:
-                    attn_metadata.qo_indptr_gpu = qo_indptr_cpu.to(self.device)
+                    attn_metadata.qo_indptr_gpu = qo_indptr_cpu.to(
+                        self.device, non_blocking=True)
                     attn_metadata.paged_kv_indptr_gpu = paged_kv_indptr_cpu.to(
-                        self.device)
+                        self.device, non_blocking=True)
 
             if num_decodes > 0:
                 pure_decode = num_prefills == 0