[Quantization] Fp8 Channelwise Dynamic Per Token GroupedGEMM #15587

robertgshaw2-redhat · 2025-03-27T01:45:29Z

SUMMARY:

follow on to @ElizaWszola's CUTLASS FP8 Grouped GEMM
enables support for channelwise scales and per token activations

LM EVAL:

channelwise, dynamic per token

lm-eval --model vllm --model_args "pretrained=nm-testing/Mixtral-8x7B-Instruct-v0.1-FP8" --tasks gsm8k --batch_size "auto"

> |Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
> |-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
> |gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.655|±  |0.0131|
> |     |       |strict-match    |     5|exact_match|↑  |0.652|±  |0.0131|

static per tensor

lm-eval --model vllm --model_args "pretrained=nm-testing/Mixtral-8x7B-Instruct-v0.1-FP8" --tasks gsm8k --batch_size "auto"

> |Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
> |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
> |gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.6406|±  |0.0132|
> |     |       |strict-match    |     5|exact_match|↑  |0.6376|±  |0.0132|

Signed-off-by: ElizaWszola <[email protected]>

Co-authored-by: Lucas Wilkinson <[email protected]> Signed-off-by: ElizaWszola <[email protected]>

Signed-off-by: ElizaWszola <[email protected]>

…of tensors Signed-off-by: ElizaWszola <[email protected]>

Signed-off-by: ElizaWszola <[email protected]>

Signed-off-by: [email protected] <[email protected]>

github-actions · 2025-03-27T01:45:38Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

Signed-off-by: [email protected] <[email protected]>

robertgshaw2-redhat · 2025-03-27T01:51:26Z

vllm/model_executor/layers/fused_moe/layer.py

            ]
        ]

-    def _load_fp8_scale(self, param: torch.nn.Parameter,


this is dead code (its not used in the codebase)

tlrmchlsmth · 2025-03-27T03:01:23Z

vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe.py

+        per_tensor = (self.weight_quant.strategy == QuantizationStrategy.TENSOR
+                      and self.input_quant.strategy
+                      == QuantizationStrategy.TENSOR)
+        per_channel = (
+            self.weight_quant.strategy == QuantizationStrategy.CHANNEL
+            and self.input_quant.strategy == QuantizationStrategy.TOKEN)


we can mix and match

ok, will change this in a FUP

…oject#15587) Signed-off-by: ElizaWszola <[email protected]> Signed-off-by: ElizaWszola <[email protected]> Signed-off-by: [email protected] <[email protected]> Co-authored-by: ElizaWszola <[email protected]> Co-authored-by: Lucas Wilkinson <[email protected]> Co-authored-by: ElizaWszola <[email protected]> Signed-off-by: xinyuxiao <[email protected]>

…oject#15587) Signed-off-by: ElizaWszola <[email protected]> Signed-off-by: ElizaWszola <[email protected]> Signed-off-by: [email protected] <[email protected]> Co-authored-by: ElizaWszola <[email protected]> Co-authored-by: Lucas Wilkinson <[email protected]> Co-authored-by: ElizaWszola <[email protected]> Signed-off-by: Louis Ulmer <[email protected]>

…oject#15587) Signed-off-by: ElizaWszola <[email protected]> Signed-off-by: ElizaWszola <[email protected]> Signed-off-by: [email protected] <[email protected]> Co-authored-by: ElizaWszola <[email protected]> Co-authored-by: Lucas Wilkinson <[email protected]> Co-authored-by: ElizaWszola <[email protected]>

…oject#15587) Signed-off-by: ElizaWszola <[email protected]> Signed-off-by: ElizaWszola <[email protected]> Signed-off-by: [email protected] <[email protected]> Co-authored-by: ElizaWszola <[email protected]> Co-authored-by: Lucas Wilkinson <[email protected]> Co-authored-by: ElizaWszola <[email protected]> Signed-off-by: Mu Huai <[email protected]>

ElizaWszola and others added 30 commits December 6, 2024 14:36

Cutlass grouped gemm files

1825ef8

Signed-off-by: ElizaWszola <[email protected]>

runs, bad result

5fd48e5

Signed-off-by: ElizaWszola <[email protected]>

A little closer to working

d5942cf

Signed-off-by: ElizaWszola <[email protected]>

Working for identical sizes

c570c69

Signed-off-by: ElizaWszola <[email protected]>

Grouped gemm working

6ed63f2

Co-authored-by: Lucas Wilkinson <[email protected]> Signed-off-by: ElizaWszola <[email protected]>

Small cleanup

e2b1fc0

Signed-off-by: ElizaWszola <[email protected]>

Merge branch 'main' into grouped-gemm-with-group-id

dd163f5

Signed-off-by: ElizaWszola <[email protected]>

Benchmark grouped cutlass against bfloat16 torch.mm

acfd3ef

Signed-off-by: ElizaWszola <[email protected]>

Merge branch 'main' into grouped-gemm-with-group-id

c6231b6

Signed-off-by: ElizaWszola <[email protected]>

Start working on fused moe cutlass implementation

f1a5666

Signed-off-by: ElizaWszola <[email protected]>

Working halfway

6414e31

Signed-off-by: ElizaWszola <[email protected]>

working mul test but the topk_weights are not yet included in kernel

67e2dd4

Signed-off-by: ElizaWszola <[email protected]>

cleaned up cutlass moe test, fixes

6523529

Signed-off-by: ElizaWszola <[email protected]>

benchmark fused

b302d98

Signed-off-by: ElizaWszola <[email protected]>

pass input as one tensor with an array of offsets rather than a list …

342d1a4

…of tensors Signed-off-by: ElizaWszola <[email protected]>

Using tensors rather than tensor lists works with test_cutlass test

7549e3d

Signed-off-by: ElizaWszola <[email protected]>

Merge branch 'main' into grouped-gemm-with-group-id

64c2a68

Signed-off-by: ElizaWszola <[email protected]>

cleanup, add import

1ea7874

Signed-off-by: ElizaWszola <[email protected]>

working fused op

d608164

Signed-off-by: ElizaWszola <[email protected]>

benchmark, create strides directly on device, small name refactor

286f6c8

Signed-off-by: ElizaWszola <[email protected]>

works with cuda graphs

b6867bb

Signed-off-by: ElizaWszola <[email protected]>

move stride tensor creation outside c++ code, cleanup

df04bc0

Signed-off-by: ElizaWszola <[email protected]>

cleanup benchmark

88c7134

Signed-off-by: ElizaWszola <[email protected]>

profile

02e1d4e

Signed-off-by: ElizaWszola <[email protected]>

tuned shapes, fix

1d9c429

Signed-off-by: ElizaWszola <[email protected]>

Merge branch 'main' into grouped-gemm-with-group-id

b824ad2

Signed-off-by: ElizaWszola <[email protected]>

Performance, add channelwise scales everywhere

ae90eee

Signed-off-by: ElizaWszola <[email protected]>

name fix

f191b35

Signed-off-by: ElizaWszola <[email protected]>

Merge branch 'main' into grouped-gemm-with-group-id

22d4f7b

perf improvements in data preparation

51941ff

Signed-off-by: ElizaWszola <[email protected]>

ElizaWszola and others added 10 commits March 24, 2025 17:05

Fixes

5ad4b0b

Signed-off-by: ElizaWszola <[email protected]>

format utils.py

41eb522

Signed-off-by: ElizaWszola <[email protected]>

Merge branch 'main' into grouped-gemm-with-group-id

f5b5c7d

Signed-off-by: ElizaWszola <[email protected]>

Make handling of both input scales consistent in the code

c6076b3

Signed-off-by: ElizaWszola <[email protected]>

Fix handling optional vals

c8f1567

Signed-off-by: ElizaWszola <[email protected]>

feedback: version checks, file structure

96296cb

Signed-off-by: ElizaWszola <[email protected]>

Change cmake flag, remove unused code

3977d67

Signed-off-by: ElizaWszola <[email protected]>

update kernel run conditions in scaled_mm_entry.cu

83ee170

Signed-off-by: ElizaWszola <[email protected]>

added channelwise, dynamic per token

fbe2b80

Signed-off-by: [email protected] <[email protected]>

updated

e0af782

Signed-off-by: [email protected] <[email protected]>

robertgshaw2-redhat requested review from mgoin and tlrmchlsmth as code owners March 27, 2025 01:45

mergify bot added the documentation Improvements or additions to documentation label Mar 27, 2025

updated

e0bae3c

Signed-off-by: [email protected] <[email protected]>

robertgshaw2-redhat commented Mar 27, 2025

View reviewed changes

tlrmchlsmth reviewed Mar 27, 2025

View reviewed changes

tlrmchlsmth approved these changes Mar 27, 2025

View reviewed changes

robertgshaw2-redhat enabled auto-merge (squash) March 27, 2025 03:34

github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Mar 27, 2025

robertgshaw2-redhat merged commit 43ed414 into vllm-project:main Mar 27, 2025
44 of 46 checks passed

ckhordiasma mentioned this pull request Apr 17, 2025

[do not merge] pr test for nm changes into 2.20 red-hat-data-services/vllm#107

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[Quantization] Fp8 Channelwise Dynamic Per Token GroupedGEMM #15587

[Quantization] Fp8 Channelwise Dynamic Per Token GroupedGEMM #15587

Uh oh!

robertgshaw2-redhat commented Mar 27, 2025 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented Mar 27, 2025

Uh oh!

robertgshaw2-redhat Mar 27, 2025 •

edited

Loading

Uh oh!

tlrmchlsmth Mar 27, 2025

Uh oh!

robertgshaw2-redhat Mar 27, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

[Quantization] Fp8 Channelwise Dynamic Per Token GroupedGEMM #15587

[Quantization] Fp8 Channelwise Dynamic Per Token GroupedGEMM #15587

Uh oh!

Conversation

robertgshaw2-redhat commented Mar 27, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 27, 2025

Uh oh!

robertgshaw2-redhat Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tlrmchlsmth Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

robertgshaw2-redhat Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

robertgshaw2-redhat commented Mar 27, 2025 •

edited by github-actions bot

Loading

robertgshaw2-redhat Mar 27, 2025 •

edited

Loading