`assemble_global_tensor` is extremely slow

[casteryh]

I believe this tensor is < 512MB in size (on CPU).

[LucasLLC]

@yuxuanchen1997, @kaiyuan-li and I spent some time debugging this. A lot of the "slowness" comes from copies necessary to re-assemble the entire tensor. This is hard to avoid, even if we use torch primitives like stack.

We will propose an experimental "defrag" endpoint, so we can control when this happens.

Fwiw @kaiyuan-li benchmarked assemble global tensor time for qwen models as ~20% of total latency