[Feature]: Extend QuantFP8 to support per-token-group quantization

[ProExpertProg]

🚀 The feature, motivation and pitch

Currently, group quantization is handled by a per_token_group_quant_fp8 custom CUDA kernel (with a Triton kernel fallback). We should fold this functionality into QuantFP8 to allow easier dispatching between CUDA, Triton, and torch implementations, automatic Inductor fusion, and easier custom op fusion.

Alternatives

No response

Additional context

This is related and complementary to #20711.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[harlanv24]

I'd like to take a stab at this one

[ProExpertProg]

@harlanv24 great, let me know if you need any help!

[zhihengy]

I am not sure about the current state of this issue but can I also try this issue? @ProExpertProg

[ProExpertProg]

@zhihengy yeah feel free to start working on this and put up a PR.

[ProExpertProg]

#24342 is the most promising approach yet, but it doesn't address the torch implementation aspect of this issue, so that's still available to be worked on for now!

[harlanv24]

I was wondering this too while working. I’m pretty new to this kind if development, is the torch implementation aspect just doing FP8 quantization if CUDA/Triton cannot be uses? E.g. using CPU? Or am I missing something. Thanks for any tips.

…

________________________________ From: Luka Govedič ***@***.***> Sent: Friday, September 5, 2025 2:10:10 PM To: vllm-project/vllm ***@***.***> Cc: Harlan Verthein ***@***.***>; Mention ***@***.***> Subject: Re: [vllm-project/vllm] [Feature]: Extend QuantFP8 to support per-token-group quantization (Issue #24185) [https://avatars.githubusercontent.com/u/11367180?s=20&v=4]ProExpertProg left a comment (vllm-project/vllm#24185)<#24185 (comment)> #24342<#24342> is the most promising approach yet, but it doesn't address the torch implementation aspect of this issue, so that's still available to be worked on for now! — Reply to this email directly, view it on GitHub<#24185 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AH3XMVMU2ZROQ3LRICWF5EL3RH33FAVCNFSM6AAAAACFRLM7S6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTENJZG42TMMBSGE>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[ProExpertProg]

@harlanv24 yeah it means it's supported on non-cuda platforms, but it also helps when using torch.compile, as it can fuse quantization onto other ops automatically.