Skip to content

[Attention] FlashAttn MLA #14258

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 49 commits into from
Sep 4, 2025
Merged

[Attention] FlashAttn MLA #14258

merged 49 commits into from
Sep 4, 2025

Conversation

LucasWilkinson
Copy link
Collaborator

@LucasWilkinson LucasWilkinson commented Mar 5, 2025 •
edited by github-actions bot
Loading

Use latest FlashAttention code to compute decode MQA in MLA

VLLM_USE_V1=0 lm_eval --model vllm --model_args pretrained=deepseek-ai/DeepSeek-V2-Lite-Chat,tensor_parallel_size=2,dtype=auto,gpu_memory_utilization=0.9,trust_remote_code=True,max_model_len=16384 --task gsm8k --num_fewshot 5  --batch_size auto

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.6725|±  |0.0129|
|     |       |strict-match    |     5|exact_match|↑  |0.6657|±  |0.0130|

Co-authored-by: @MatthewBonanni

@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 5, 2025

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

This was referenced Mar 5, 2025
Merged
Merged
@LucasWilkinson LucasWilkinson mentioned this pull request Mar 13, 2025
Merged
@mergify Mergify
Copy link

mergify bot commented Mar 26, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Mar 26, 2025
@mergify mergify bot removed the needs-rebase label Apr 18, 2025
@mergify Mergify
Copy link

mergify bot commented Apr 22, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Apr 22, 2025
@LucasWilkinson LucasWilkinson mentioned this pull request Jul 31, 2025
Open
@mergify mergify bot added rocm Related to AMD ROCm and removed needs-rebase labels Aug 25, 2025
LucasWilkinson and others added 14 commits August 25, 2025 20:07
@LucasWilkinson @MatthewBonanni
sync with amd, support v1
5ea3788 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson @MatthewBonanni
fix IMA
d048631 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson @MatthewBonanni
bugfix
e09841c 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson @MatthewBonanni
working
52e7234 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson @MatthewBonanni
cleanup
4604201 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson @MatthewBonanni
fa MLA
f9f3e3e 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson @MatthewBonanni
commit wip
b3b060b 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson @MatthewBonanni
cleanup
e286de8 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson @MatthewBonanni
fix
27a2cd2 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson @MatthewBonanni
move files
9165af3 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson @MatthewBonanni
fix up
d056efd 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson @MatthewBonanni
v0 support + decode threshold
ac4c624 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson @MatthewBonanni
v0 fix
1f6bb3d 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson @MatthewBonanni
fix
73c8736 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson
missing line
4f29ce1 
Signed-off-by: Lucas Wilkinson <[email protected]>
@LucasWilkinson LucasWilkinson enabled auto-merge (squash) August 28, 2025 14:34
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 28, 2025
auto-merge was automatically disabled August 28, 2025 18:56

Head branch was pushed to by a user without write access

@LucasWilkinson LucasWilkinson enabled auto-merge (squash) August 28, 2025 19:03
@MatthewBonanni MatthewBonanni mentioned this pull request Aug 29, 2025
Merged
5 tasks
@vllm-bot vllm-bot merged commit 402759d into vllm-project:main Sep 4, 2025
68 of 71 checks passed
@MatthewBonanni MatthewBonanni deleted the lwilkinson/fa-mla branch September 4, 2025 13:35
@MatthewBonanni MatthewBonanni restored the lwilkinson/fa-mla branch September 4, 2025 14:18
@MatthewBonanni MatthewBonanni mentioned this pull request Sep 5, 2025
Merged
4 tasks
@MatthewBonanni MatthewBonanni deleted the lwilkinson/fa-mla branch September 8, 2025 19:05
eicherseiji pushed a commit to eicherseiji/vllm that referenced this pull request Sep 9, 2025
@LucasWilkinson @MatthewBonanni @eicherseiji
[Attention] FlashAttn MLA (vllm-project#14258)
9241e3a 
Signed-off-by: Lucas Wilkinson <[email protected]>
Signed-off-by: Lucas Wilkinson <[email protected]>
Signed-off-by: Matthew Bonanni <[email protected]>
Co-authored-by: Matthew Bonanni <[email protected]>
Co-authored-by: Matthew Bonanni <[email protected]>
FeiDaLI pushed a commit to FeiDaLI/vllm that referenced this pull request Sep 25, 2025
@LucasWilkinson @MatthewBonanni @FeiDaLI
[Attention] FlashAttn MLA (vllm-project#14258)
20d8a1e 
Signed-off-by: Lucas Wilkinson <[email protected]>
Signed-off-by: Lucas Wilkinson <[email protected]>
Signed-off-by: Matthew Bonanni <[email protected]>
Co-authored-by: Matthew Bonanni <[email protected]>
Co-authored-by: Matthew Bonanni <[email protected]>
xuebwang-amd pushed a commit to xuebwang-amd/vllm that referenced this pull request Oct 10, 2025
@LucasWilkinson @MatthewBonanni @xuebwang-amd
[Attention] FlashAttn MLA (vllm-project#14258)
ca0a7b5 
Signed-off-by: Lucas Wilkinson <[email protected]>
Signed-off-by: Lucas Wilkinson <[email protected]>
Signed-off-by: Matthew Bonanni <[email protected]>
Co-authored-by: Matthew Bonanni <[email protected]>
Co-authored-by: Matthew Bonanni <[email protected]>
Signed-off-by: xuebwang-amd <[email protected]>
@MatthewBonanni MatthewBonanni mentioned this pull request Oct 14, 2025
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@robertgshaw2-redhat robertgshaw2-redhat robertgshaw2-redhat approved these changes

@gshtras gshtras Awaiting requested review from gshtras gshtras is a code owner

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner

@njhill njhill Awaiting requested review from njhill

@ywang96 ywang96 Awaiting requested review from ywang96

@comaniac comaniac Awaiting requested review from comaniac

@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat

@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth is a code owner

@yewentao256 yewentao256 Awaiting requested review from yewentao256 yewentao256 is a code owner

+1 more reviewer

@MatthewBonanni MatthewBonanni MatthewBonanni left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ci/build ready ONLY add when PR is ready to merge/full CI is needed rocm Related to AMD ROCm v1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@LucasWilkinson @MatthewBonanni @robertgshaw2-redhat @vllm-bot