[Multimodal][Speculative Decoding]Eagle Eagle3 mm support, enablement on qwen2.5vl #22872

david6666666 · 2025-08-14T03:28:49Z

Purpose

follow #20788 Eagle Eagle3 mm support, enablement on qwen2.5vl
model is https://huggingface.co/Rayzl/qwen2.5-vl-7b-eagle3-sgl

Test Plan

Test Result

vllm serve \
    /workspace/models/Qwen2.5-VL-7B-Instruct \
    --port 5580 --host 0.0.0.0 \
    --max-num-seqs 128 --dtype bfloat16 --max-model-len=8192  \
    --no-enable-prefix-caching --trust-remote-code -tp 1\
    --speculative-config '{"method": "eagle3", "model": "/workspace/models/qwen2.5-vl-7b-eagle3-sgl", "prefill_token_shift": false, "num_speculative_tokens": 3, "draft_tensor_parallel_size": 1, "max_model_len": 8192}' \
    --num-lookahead-slots=3 \
    --gpu-memory-utilization=0.93

client

vllm bench serve \
--backend openai-chat \
--endpoint-type openai-chat \
--port 5580 \
--host 127.0.0.1 \
--endpoint /v1/chat/completions \
--model /home/models/Qwen2.5-VL-7B-Instruct \
--dataset-name hf \
--dataset-path Lin-Chen/MMStar \
--hf-split val \
--num-prompts 50 \
--max-concurrency 2 \
--temperature 0 \
--top-p 1.0 \
--seed 123

e2e speed up 1.3x
# eagle3
============ Serving Benchmark Result ============
Successful requests:                     50        
Maximum request concurrency:             2         
Benchmark duration (s):                  32.11     
Total input tokens:                      1226      
Total generated tokens:                  3985      
Request throughput (req/s):              1.56      
Output token throughput (tok/s):         124.09    
Peak output token throughput (tok/s):    82.00     
Peak concurrent requests:                6.00      
Total Token throughput (tok/s):          162.26    
---------------Time to First Token----------------
Mean TTFT (ms):                          50.02     
Median TTFT (ms):                        49.18     
P99 TTFT (ms):                           71.19     
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          14.07     
Median TPOT (ms):                        14.27     
P99 TPOT (ms):                           19.15     
---------------Inter-token Latency----------------
Mean ITL (ms):                           24.34     
Median ITL (ms):                         24.75     
P99 ITL (ms):                            25.99     
==================================================


# without eagle3
tip: install termplotlib and gnuplot to plot the metrics
============ Serving Benchmark Result ============
Successful requests:                     50        
Maximum request concurrency:             2         
Benchmark duration (s):                  44.76     
Total input tokens:                      1226      
Total generated tokens:                  4237      
Request throughput (req/s):              1.12      
Output token throughput (tok/s):         94.66     
Peak output token throughput (tok/s):    99.00     
Peak concurrent requests:                5.00      
Total Token throughput (tok/s):          122.05    
---------------Time to First Token----------------
Mean TTFT (ms):                          43.05     
Median TTFT (ms):                        41.76     
P99 TTFT (ms):                           62.82     
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          20.48     
Median TPOT (ms):                        20.49     
P99 TPOT (ms):                           20.64     
---------------Inter-token Latency----------------
Mean ITL (ms):                           20.24     
Median ITL (ms):                         20.46     
P99 ITL (ms):                            21.60     
==================================================

(Optional) Documentation Update

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

gemini-code-assist

Code Review

This pull request adds support for Eagle and Eagle3 speculative decoding for the Qwen2.5-VL multimodal model. The changes include adding new model files for the Eagle and Eagle3 variants, updating model registries, and modifying tests. I found one issue related to a buggy condition in the weight loading logic for the Eagle model which could lead to incorrect behavior. My feedback addresses this.

vllm/model_executor/models/qwen2_5_vl_eagle.py

github-actions · 2025-08-14T03:54:38Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

DarkLight1337 · 2025-08-14T04:46:57Z

tests/v1/e2e/test_spec_decode.py

@@ -183,6 +191,9 @@ def test_eagle_correctness(

        method, model_name, spec_model_name, tp_size = model_setup

+        if "Qwen2.5-VL" in model_name and attn_backend == "TREE_ATTN":
+            pytest.skip("TREE ATTN not support Qwen2.5-VL Model yet")
+        print(f"model_setup={model_setup}")


Suggested change

print(f"model_setup={model_setup}")

DarkLight1337

cc @22quinn @morgendave can you help review?

Signed-off-by: Junhong <[email protected]>

22quinn · 2025-08-16T06:43:27Z

cc @22quinn @morgendave can you help review?

cc spec decode experts @zixi-qi @charlotte12l

mergify · 2025-08-20T20:14:32Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @david6666666.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Junhong <[email protected]>

vllm/v1/spec_decode/eagle.py

Signed-off-by: Junhong Liu <[email protected]>

Signed-off-by: Junhong <[email protected]>

vllm/v1/spec_decode/eagle.py

Signed-off-by: Junhong <[email protected]>

vllm/v1/worker/gpu_model_runner.py

Signed-off-by: Junhong <[email protected]>

DarkLight1337

Thanks for your patience!

Signed-off-by: Junhong <[email protected]>

DarkLight1337 · 2025-09-27T00:06:01Z

PTAL at the failing test

Signed-off-by: Junhong <[email protected]>

mergify · 2025-09-27T01:30:09Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @david6666666.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Junhong Liu <[email protected]>

LJH-LBJ · 2025-09-27T01:57:26Z

PTAL at the failing test

skip the test_eagle_correctness, because this flash attention build does not support headdim not being a multiple of 32

taoyun951753 · 2025-09-29T10:15:02Z

The online service, deployed with the Qwen2.5-VL-7B model, encountered a corner case that caused the service to crash. The input_ids processed by the propose function in eagle.py generated an image_token_id, resulting in a shape mismatch with mm_embeds.

taoyun951753 · 2025-09-29T10:19:26Z

mm embds shape: [torch.Size([546, 3584])]
before propose input ids: target token ids: tensor([151644, 8948, 198, 2610, 525, 264, 10950, 17847, 13,
151645, 198, 151644, 872, 198, 151652, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151653, 53481, 41683, 17254, 9370, 45930,
102073, 43815, 8997, 101140, 37945, 104317, 102650, 15946, 64471,
101114, 1773, 62244, 102650, 15946, 101475, 37945, 53481, 102650,
101047, 102122, 24968, 62244, 102650, 15946, 101114, 37945, 53481,
102650, 15946, 104013, 109318, 101070, 8997, 14880, 23031, 2236,
68805, 99553, 7, 104346, 3837, 100714, 53481, 3837, 99226,
98237, 53481, 8, 101124, 66017, 3837, 90919, 104346, 14880,
11622, 2073, 106468, 5373, 101977, 5373, 42140, 99605, 854,
104787, 1773, 101475, 102122, 15946, 99226, 98237, 53481, 91680,
102298, 104766, 5373, 44091, 3837, 100714, 53481, 46448, 108593,
102298, 102122, 106336, 49567, 104449, 1773, 101114, 102122, 15946,
99226, 98237, 53481, 91680, 102298, 104013, 5373, 104766, 5373,
102196, 3837, 100714, 53481, 46448, 108593, 102298, 104013, 104363,
5373, 104766, 99719, 49567, 100405, 104449, 1773, 151645, 198,
151644, 77091, 198, 57218, 112531, 99194, 38035, 17447, 24562,
100544, 3837, 104013, 109612, 18830, 17323, 71, 3022, 43,
46944, 112531, 99194, 38035, 17447], device='cuda:0',
dtype=torch.int32), next token ids: tensor([ 4913, 111413, 110607, 100010, 105611], device='cuda:0',
dtype=torch.int32) ,mm_embeds shape: [torch.Size([546, 3584])]
propose input ids: tensor([ 8948, 198, 2610, 525, 264, 10950, 17847, 13, 151645,
198, 151644, 872, 198, 151652, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
151655, 151655, 151653, 53481, 41683, 17254, 9370, 45930, 102073,
43815, 8997, 101140, 37945, 104317, 102650, 15946, 64471, 101114,
1773, 62244, 102650, 15946, 101475, 37945, 53481, 102650, 101047,
102122, 24968, 62244, 102650, 15946, 101114, 37945, 53481, 102650,
15946, 104013, 109318, 101070, 8997, 14880, 23031, 2236, 68805,
99553, 7, 104346, 3837, 100714, 53481, 3837, 99226, 98237,
53481, 8, 101124, 66017, 3837, 90919, 104346, 14880, 11622,
2073, 106468, 5373, 101977, 5373, 42140, 99605, 854, 104787,
1773, 101475, 102122, 15946, 99226, 98237, 53481, 91680, 102298,
104766, 5373, 44091, 3837, 100714, 53481, 46448, 108593, 102298,
102122, 106336, 49567, 104449, 1773, 101114, 102122, 15946, 99226,
98237, 53481, 91680, 102298, 104013, 5373, 104766, 5373, 102196,
3837, 100714, 53481, 46448, 108593, 102298, 104013, 104363, 5373,
104766, 99719, 49567, 100405, 104449, 1773, 151645, 198, 151644,
77091, 198, 4913, 112531, 111413, 38035, 17447, 24562, 100544,
3837, 110607, 109612, 18830, 100010, 71, 3022, 43, 46944,
105611, 99194, 38035, 17447, 151655], device='cuda:0',
dtype=torch.int32), mm embeds shape :[torch.Size([546, 3584])]
utils.py: is_multimodal sum: 547, emb shape 0:546
masked_scatter_size_check: block: [0,0,0], thread: [0,0,0] Assertion totalElements <= srcSize failed.
ERROR 09-29 03:07:32 [dump_input.py:69] Dumping input data for V1 LLM engine (v0.11.0rc2.dev0+g0d9fe260d.d20250926) with config: model='/lhf-model/Ipc/Qwen2.5-VL-7B-Instruct-20250829', speculative_config=SpeculativeConfig(method='eagle3', model='/mnt/sharedLLMs/qwen2.5-vl-7b-eagle3-specforge-v5-epoch1', num_spec_tokens=4), tokenizer='/lhf-model/Ipc/Qwen2.5-VL-7B-Instruct-20250829', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=base_model, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"/root/.cache/vllm/torch_compile_cache/6cd3220e9b","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2","vllm.mamba_mixer","vllm.short_conv","vllm.linear_attention","vllm.plamo2_mamba_mixer","vllm.gdn_attention"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"cudagraph_mode":1,"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"use_inductor_graph_partition":false,"pass_config":{},"max_capture_size":512,"local_cache_dir":"/root/.cache/vllm/torch_compile_cache/6cd3220e9b/rank_0_0/backbone"},
ERROR 09-29 03:07:32 [dump_input.py:76] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=chatcmpl-c0150128df3f4637864541d650a02fb9,prompt_token_ids_len=687,mm_features=[MultiModalFeatureSpec(data={'image_grid_thw': MultiModalFieldElem(modality='image', key='image_grid_thw', data=tensor([ 1, 42, 52]), field=MultiModalBatchedField()), 'pixel_values': MultiModalFieldElem(modality='image', key='pixel_values', data=tensor([[1.9297, 1.9297, 1.9297, ..., 2.1406, 2.1406, 2.1406],
ERROR 09-29 03:07:32 [dump_input.py:76] [1.9297, 1.9297, 1.9297, ..., 2.1406, 2.1406, 2.1406],
ERROR 09-29 03:07:32 [dump_input.py:76] [1.9297, 1.9297, 1.9297, ..., 2.1406, 2.1406, 2.1406],
ERROR 09-29 03:07:32 [dump_input.py:76] ...,
ERROR 09-29 03:07:32 [dump_input.py:76] [1.9297, 1.9297, 1.9297, ..., 2.1406, 2.1406, 2.1406],
ERROR 09-29 03:07:32 [dump_input.py:76] [1.9297, 1.9297, 1.9297, ..., 2.1406, 2.1406, 2.1406],
ERROR 09-29 03:07:32 [dump_input.py:76] [1.9297, 1.9297, 1.9297, ..., 2.1406, 2.1406, 2.1406]],
ERROR 09-29 03:07:32 [dump_input.py:76] dtype=torch.bfloat16), field=MultiModalFlatField(slices=[[slice(0, 2184, None)]], dim=0))}, modality='image', identifier='089654d81821ee9d7bbe74606f03e9df82789e90e7bb59dbf5bf3b1df03c527f', mm_position=PlaceholderRange(offset=15, length=546, is_embed=None))],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.05, temperature=0.0, top_p=1.0, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[151643], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=128, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, structured_outputs=None, extra_args=None),block_ids=([1299, 1300, 1301, 1302, 1303, 1304, 1305, 1306, 1307, 1308, 1309, 1310, 1311, 1312, 1313, 1314, 1315, 1316, 1317, 1318, 1319, 1320, 1321, 1322, 1323, 1324, 1325, 1326, 1327, 1328, 1329, 1330, 1331, 1332, 1333, 1334, 1335, 1336, 1337, 1338, 1339, 1340, 1341],),num_computed_tokens=0,lora_request=None,prompt_embeds_shape=None)], scheduled_cached_reqs=CachedRequestData(req_ids=['chatcmpl-0721c0b185fd419c9a4594955b125766', 'chatcmpl-c123b1c2a6124ba4b6751f9295fedb60', 'chatcmpl-aa2319efc98c4342838ea42e267ab5d2', 'chatcmpl-029d33be6527448bb50c1417a1d02b0f'], resumed_from_preemption=[false, false, false, false], new_token_ids=[], new_block_ids=[[[1297]], null, null, [[1298]]], num_computed_tokens=[3880, 899, 679, 633]), num_scheduled_tokens={chatcmpl-c0150128df3f4637864541d650a02fb9: 687, chatcmpl-0721c0b185fd419c9a4594955b125766: 5, chatcmpl-c123b1c2a6124ba4b6751f9295fedb60: 5, chatcmpl-aa2319efc98c4342838ea42e267ab5d2: 5, chatcmpl-029d33be6527448bb50c1417a1d02b0f: 5}, total_num_scheduled_tokens=707, scheduled_spec_decode_tokens={chatcmpl-029d33be6527448bb50c1417a1d02b0f: [112531, 99194, 38035, 17447], chatcmpl-0721c0b185fd419c9a4594955b125766: [100544, 3837, 104013, 109612], chatcmpl-c123b1c2a6124ba4b6751f9295fedb60: [17323, 71, 3022, 43], chatcmpl-aa2319efc98c4342838ea42e267ab5d2: [112531, 99194, 38035, 17447]}, scheduled_encoder_inputs={chatcmpl-c0150128df3f4637864541d650a02fb9: [0]}, num_common_prefix_blocks=[0], finished_req_ids=['chatcmpl-e4e8edd6366042429bfaebe5479951bb'], free_encoder_mm_hashes=['dd8c6601b7dec1227a8dbc366c398edb0c05d6a568ab4153729d1df02fe14e88'], structured_output_request_ids={}, grammar_bitmask=null, kv_connector_metadata=null)
ERROR 09-29 03:07:32 [dump_input.py:79] Dumping scheduler stats: SchedulerStats(num_running_reqs=5, num_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.26257668711656446, prefix_cache_stats=PrefixCacheStats(reset=False, requests=1, queries=687, hits=0), spec_decoding_stats=None, kv_connector_stats=None, num_corrupted_reqs=0)
ERROR 09-29 03:07:32 [core.py:710] EngineCore encountered a fatal error.
ERROR 09-29 03:07:32 [core.py:710] Traceback (most recent call last):
ERROR 09-29 03:07:32 [core.py:710] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 409, in merge_multimodal_embeddings
ERROR 09-29 03:07:32 [core.py:710] inputs_embeds.masked_scatter(is_multimodal.unsqueeze(-1),
ERROR 09-29 03:07:32 [core.py:710] torch.AcceleratorError: CUDA error: device-side assert triggered
ERROR 09-29 03:07:32 [core.py:710] Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
ERROR 09-29 03:07:32 [core.py:710]

ggg-s · 2025-09-29T10:24:21Z

mm embds shape: [torch.Size([546, 3584])]
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m before propose input ids: target token ids: tensor([151644, 8948, 198, 2610, 525, 264, 10950, 17847, 13,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151645, 198, 151644, 872, 198, 151652, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151653, 53481, 41683, 17254, 9370, 45930,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 102073, 43815, 8997, 101140, 37945, 104317, 102650, 15946, 64471,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 101114, 1773, 62244, 102650, 15946, 101475, 37945, 53481, 102650,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 101047, 102122, 24968, 62244, 102650, 15946, 101114, 37945, 53481,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 102650, 15946, 104013, 109318, 101070, 8997, 14880, 23031, 2236,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 68805, 99553, 7, 104346, 3837, 100714, 53481, 3837, 99226,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 98237, 53481, 8, 101124, 66017, 3837, 90919, 104346, 14880,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 11622, 2073, 106468, 5373, 101977, 5373, 42140, 99605, 854,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 104787, 1773, 101475, 102122, 15946, 99226, 98237, 53481, 91680,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 102298, 104766, 5373, 44091, 3837, 100714, 53481, 46448, 108593,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 102298, 102122, 106336, 49567, 104449, 1773, 101114, 102122, 15946,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 99226, 98237, 53481, 91680, 102298, 104013, 5373, 104766, 5373,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 102196, 3837, 100714, 53481, 46448, 108593, 102298, 104013, 104363,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 5373, 104766, 99719, 49567, 100405, 104449, 1773, 151645, 198,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151644, 77091, 198, 57218, 112531, 99194, 38035, 17447, 24562,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 100544, 3837, 104013, 109612, 18830, 17323, 71, 3022, 43,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 46944, 112531, 99194, 38035, 17447], device='cuda:0',
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m dtype=torch.int32), next token ids: tensor([ 4913, 111413, 110607, 100010, 105611], device='cuda:0',
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m dtype=torch.int32) ,mm_embeds shape: [torch.Size([546, 3584])]
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m propose input ids: tensor([ 8948, 198, 2610, 525, 264, 10950, 17847, 13, 151645,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 198, 151644, 872, 198, 151652, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151653, 53481, 41683, 17254, 9370, 45930, 102073,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 43815, 8997, 101140, 37945, 104317, 102650, 15946, 64471, 101114,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 1773, 62244, 102650, 15946, 101475, 37945, 53481, 102650, 101047,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 102122, 24968, 62244, 102650, 15946, 101114, 37945, 53481, 102650,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 15946, 104013, 109318, 101070, 8997, 14880, 23031, 2236, 68805,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 99553, 7, 104346, 3837, 100714, 53481, 3837, 99226, 98237,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 53481, 8, 101124, 66017, 3837, 90919, 104346, 14880, 11622,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 2073, 106468, 5373, 101977, 5373, 42140, 99605, 854, 104787,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 1773, 101475, 102122, 15946, 99226, 98237, 53481, 91680, 102298,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 104766, 5373, 44091, 3837, 100714, 53481, 46448, 108593, 102298,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 102122, 106336, 49567, 104449, 1773, 101114, 102122, 15946, 99226,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 98237, 53481, 91680, 102298, 104013, 5373, 104766, 5373, 102196,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 3837, 100714, 53481, 46448, 108593, 102298, 104013, 104363, 5373,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 104766, 99719, 49567, 100405, 104449, 1773, 151645, 198, 151644,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 77091, 198, 4913, 112531, 111413, 38035, 17447, 24562, 100544,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 3837, 110607, 109612, 18830, 100010, 71, 3022, 43, 46944,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 105611, 99194, 38035, 17447, 151655], device='cuda:0',
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m dtype=torch.int32), mm embeds shape :[torch.Size([546, 3584])]
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m utils.py: is_multimodal sum: 547, emb shape 0:546
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:402: masked_scatter_size_check: block: [0,0,0], thread: [0,0,0] Assertion totalElements <= srcSize failed.
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:69] Dumping input data for V1 LLM engine (v0.11.0rc2.dev0+g0d9fe260d.d20250926) with config: model='/lhf-model/Ipc/Qwen2.5-VL-7B-Instruct-20250829', speculative_config=SpeculativeConfig(method='eagle3', model='/mnt/sharedLLMs/qwen2.5-vl-7b-eagle3-specforge-v5-epoch1', num_spec_tokens=4), tokenizer='/lhf-model/Ipc/Qwen2.5-VL-7B-Instruct-20250829', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=base_model, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"/root/.cache/vllm/torch_compile_cache/6cd3220e9b","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2","vllm.mamba_mixer","vllm.short_conv","vllm.linear_attention","vllm.plamo2_mamba_mixer","vllm.gdn_attention"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"cudagraph_mode":1,"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"use_inductor_graph_partition":false,"pass_config":{},"max_capture_size":512,"local_cache_dir":"/root/.cache/vllm/torch_compile_cache/6cd3220e9b/rank_0_0/backbone"},
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:76] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=chatcmpl-c0150128df3f4637864541d650a02fb9,prompt_token_ids_len=687,mm_features=[MultiModalFeatureSpec(data={'image_grid_thw': MultiModalFieldElem(modality='image', key='image_grid_thw', data=tensor([ 1, 42, 52]), field=MultiModalBatchedField()), 'pixel_values': MultiModalFieldElem(modality='image', key='pixel_values', data=tensor([[1.9297, 1.9297, 1.9297, ..., 2.1406, 2.1406, 2.1406],
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:76] [1.9297, 1.9297, 1.9297, ..., 2.1406, 2.1406, 2.1406],
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:76] [1.9297, 1.9297, 1.9297, ..., 2.1406, 2.1406, 2.1406],
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:76] ...,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:76] [1.9297, 1.9297, 1.9297, ..., 2.1406, 2.1406, 2.1406],
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:76] [1.9297, 1.9297, 1.9297, ..., 2.1406, 2.1406, 2.1406],
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:76] [1.9297, 1.9297, 1.9297, ..., 2.1406, 2.1406, 2.1406]],
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:76] dtype=torch.bfloat16), field=MultiModalFlatField(slices=[[slice(0, 2184, None)]], dim=0))}, modality='image', identifier='089654d81821ee9d7bbe74606f03e9df82789e90e7bb59dbf5bf3b1df03c527f', mm_position=PlaceholderRange(offset=15, length=546, is_embed=None))],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.05, temperature=0.0, top_p=1.0, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[151643], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=128, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, structured_outputs=None, extra_args=None),block_ids=([1299, 1300, 1301, 1302, 1303, 1304, 1305, 1306, 1307, 1308, 1309, 1310, 1311, 1312, 1313, 1314, 1315, 1316, 1317, 1318, 1319, 1320, 1321, 1322, 1323, 1324, 1325, 1326, 1327, 1328, 1329, 1330, 1331, 1332, 1333, 1334, 1335, 1336, 1337, 1338, 1339, 1340, 1341],),num_computed_tokens=0,lora_request=None,prompt_embeds_shape=None)], scheduled_cached_reqs=CachedRequestData(req_ids=['chatcmpl-0721c0b185fd419c9a4594955b125766', 'chatcmpl-c123b1c2a6124ba4b6751f9295fedb60', 'chatcmpl-aa2319efc98c4342838ea42e267ab5d2', 'chatcmpl-029d33be6527448bb50c1417a1d02b0f'], resumed_from_preemption=[false, false, false, false], new_token_ids=[], new_block_ids=[[[1297]], null, null, [[1298]]], num_computed_tokens=[3880, 899, 679, 633]), num_scheduled_tokens={chatcmpl-c0150128df3f4637864541d650a02fb9: 687, chatcmpl-0721c0b185fd419c9a4594955b125766: 5, chatcmpl-c123b1c2a6124ba4b6751f9295fedb60: 5, chatcmpl-aa2319efc98c4342838ea42e267ab5d2: 5, chatcmpl-029d33be6527448bb50c1417a1d02b0f: 5}, total_num_scheduled_tokens=707, scheduled_spec_decode_tokens={chatcmpl-029d33be6527448bb50c1417a1d02b0f: [112531, 99194, 38035, 17447], chatcmpl-0721c0b185fd419c9a4594955b125766: [100544, 3837, 104013, 109612], chatcmpl-c123b1c2a6124ba4b6751f9295fedb60: [17323, 71, 3022, 43], chatcmpl-aa2319efc98c4342838ea42e267ab5d2: [112531, 99194, 38035, 17447]}, scheduled_encoder_inputs={chatcmpl-c0150128df3f4637864541d650a02fb9: [0]}, num_common_prefix_blocks=[0], finished_req_ids=['chatcmpl-e4e8edd6366042429bfaebe5479951bb'], free_encoder_mm_hashes=['dd8c6601b7dec1227a8dbc366c398edb0c05d6a568ab4153729d1df02fe14e88'], structured_output_request_ids={}, grammar_bitmask=null, kv_connector_metadata=null)
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:79] Dumping scheduler stats: SchedulerStats(num_running_reqs=5, num_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.26257668711656446, prefix_cache_stats=PrefixCacheStats(reset=False, requests=1, queries=687, hits=0), spec_decoding_stats=None, kv_connector_stats=None, num_corrupted_reqs=0)
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [core.py:710] EngineCore encountered a fatal error.
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [core.py:710] Traceback (most recent call last):
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [core.py:710] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 409, in merge_multimodal_embeddings
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [core.py:710] inputs_embeds.masked_scatter(is_multimodal.unsqueeze(-1),
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [core.py:710] torch.AcceleratorError: CUDA error: device-side assert triggered
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [core.py:710] Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [core.py:710]

Please print the error logs from the very beginning.

make placeholders and vision patches the same length (trim to match) inside merge_multimodal_embeddings

DarkLight1337 · 2025-09-29T10:48:08Z

This corner case should be mitigated by #16229, which was merged recently

… on qwen2.5vl (vllm-project#22872) Signed-off-by: Junhong <[email protected]> Signed-off-by: Junhong Liu <[email protected]> Co-authored-by: Junhong <[email protected]> Co-authored-by: LJH-LBJ <[email protected]>

… on qwen2.5vl (#22872) Signed-off-by: Junhong <[email protected]> Signed-off-by: Junhong Liu <[email protected]> Co-authored-by: Junhong <[email protected]> Co-authored-by: LJH-LBJ <[email protected]> Signed-off-by: yewentao256 <[email protected]>

benchislett · 2025-10-08T13:51:06Z

Does anyone know why this PR removes torch compile support for llama_eagle3?

taoyun951753 · 2025-10-09T08:28:52Z

The qwen2.5-vl eagle3 model encounters an error when using @support_torch_compile

Traceback (most recent call last):
(EngineCore_DP0 pid=8139) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=8139) self.run()
(EngineCore_DP0 pid=8139) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=8139) self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 716, in run_engine_core
(EngineCore_DP0 pid=8139) raise e
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 703, in run_engine_core
(EngineCore_DP0 pid=8139) engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 499, in init
(EngineCore_DP0 pid=8139) super().init(vllm_config, executor_class, log_stats,
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 93, in init
(EngineCore_DP0 pid=8139) self._initialize_kv_caches(vllm_config)
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 208, in _initialize_kv_caches
(EngineCore_DP0 pid=8139) self.model_executor.initialize_from_config(kv_cache_configs)
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 75, in initialize_from_config
(EngineCore_DP0 pid=8139) self.collective_rpc("compile_or_warm_up_model")
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc
(EngineCore_DP0 pid=8139) return [run_method(self.driver_worker, method, args, kwargs)]
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/utils/init.py", line 3121, in run_method
(EngineCore_DP0 pid=8139) return func(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 344, in compile_or_warm_up_model
(EngineCore_DP0 pid=8139) cuda_graph_memory_bytes = self.model_runner.capture_model()
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3520, in capture_model
(EngineCore_DP0 pid=8139) self._capture_cudagraphs(
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3597, in _capture_cudagraphs
(EngineCore_DP0 pid=8139) self._dummy_run(num_tokens,
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=8139) return func(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3240, in dummy_run
(EngineCore_DP0 pid=8139) self.drafter.dummy_run(num_tokens,use_cudagraphs=cudagraph_runtime_mode!=CUDAGraphMode.NONE)
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/utils/contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=8139) return func(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/spec_decode/eagle.py", line 966, in dummy_run
(EngineCore_DP0 pid=8139) self.model(
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in wrapped_call_impl
(EngineCore_DP0 pid=8139) return self.call_impl(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in call_impl
(EngineCore_DP0 pid=8139) return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2_5_vl_eagle3.py", line 258, in forward
(EngineCore_DP0 pid=8139) return self.model(input_ids, positions, hidden_states, inputs_embeds)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 317, in call
(EngineCore_DP0 pid=8139) model_output = self.forward(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2_5_vl_eagle3.py", line 154, in forward
(EngineCore_DP0 pid=8139) def forward(
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/dynamo/eval_frame.py", line 375, in call
(EngineCore_DP0 pid=8139) return super().call(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in wrapped_call_impl
(EngineCore_DP0 pid=8139) return self.call_impl(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in call_impl
(EngineCore_DP0 pid=8139) return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/dynamo/eval_frame.py", line 929, in fn
(EngineCore_DP0 pid=8139) return fn(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 848, in call_wrapped
(EngineCore_DP0 pid=8139) return self.wrapped_call(self, *args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 424, in call
(EngineCore_DP0 pid=8139) raise e
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 411, in call
(EngineCore_DP0 pid=8139) return super(self.cls, obj).call(*args, **kwargs) # type: ignore[misc]
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in wrapped_call_impl
(EngineCore_DP0 pid=8139) return self.call_impl(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in call_impl
(EngineCore_DP0 pid=8139) return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "<eval_with_key>.271", line 18, in forward
(EngineCore_DP0 pid=8139) submod_0 = self.submod_0(l_input_ids, s72, l_self_modules_embed_tokens_parameters_weight, l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight, l_hidden_states, l_self_modules_layers_modules_0_modules_hidden_norm_parameters_weight, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_weight, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache, l_positions, s18); l_input_ids = l_self_modules_embed_tokens_parameters_weight = l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight = l_self_modules_layers_modules_0_modules_hidden_norm_parameters_weight = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_weight = l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache = l_positions = s18 = None
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/cuda_graph.py", line 121, in call
(EngineCore_DP0 pid=8139) return self.runnable(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/piecewise_backend.py", line 96, in call
(EngineCore_DP0 pid=8139) return self.compiled_graph_for_general_shape(*args)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 929, in _fn
(EngineCore_DP0 pid=8139) return fn(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/aot_autograd.py", line 1241, in forward
(EngineCore_DP0 pid=8139) return compiled_fn(full_args)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 384, in runtime_wrapper
(EngineCore_DP0 pid=8139) all_outs = call_func_at_runtime_with_args(
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/utils.py", line 126, in call_func_at_runtime_with_args
(EngineCore_DP0 pid=8139) out = normalize_as_list(f(args))
(EngineCore_DP0 pid=8139) ^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 750, in inner_fn
(EngineCore_DP0 pid=8139) outs = compiled_fn(args)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 556, in wrapper
(EngineCore_DP0 pid=8139) return compiled_fn(runtime_args)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/output_code.py", line 584, in call
(EngineCore_DP0 pid=8139) return self.current_callable(inputs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/utils.py", line 2716, in run
(EngineCore_DP0 pid=8139) out = model(new_inputs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/root/.cache/vllm/torch_compile_cache/8304a885d2/rank_0_0/inductor_cache/2i/c2iggmzrq2sxtsm37f72hc22obqnk3swtexiwdxv5crnyca2zvan.py", line 622, in call
(EngineCore_DP0 pid=8139) assert_size_stride(arg8_1, (3, s18), (s18, 1))
(EngineCore_DP0 pid=8139) AssertionError: expected size 3==3, stride 2048==1536 at dim=0
(EngineCore_DP0 pid=8139) This error most often comes from a incorrect fake (aka meta) kernel for a custom op.

… on qwen2.5vl (vllm-project#22872) Signed-off-by: Junhong <[email protected]> Signed-off-by: Junhong Liu <[email protected]> Co-authored-by: Junhong <[email protected]> Co-authored-by: LJH-LBJ <[email protected]> Signed-off-by: xuebwang-amd <[email protected]>

david6666666 requested review from DarkLight1337, WoosukKwon, alexm-redhat, comaniac, njhill, robertgshaw2-redhat, sighingnow and ywang96 as code owners August 14, 2025 03:28

mergify bot added new-model Requests to new models qwen Related to Qwen models speculative-decoding v1 labels Aug 14, 2025

gemini-code-assist bot reviewed Aug 14, 2025

View reviewed changes

vllm/model_executor/models/qwen2_5_vl_eagle.py Outdated Show resolved Hide resolved

DarkLight1337 reviewed Aug 14, 2025

View reviewed changes

LJH-LBJ force-pushed the Eagle-mulitmodal-support-Qwen2.5vl branch from af49ffc to 9d06a8d Compare August 14, 2025 05:11

[V1] support eagle and eagle3 for qwen2_5vl

8874e16

Signed-off-by: Junhong <[email protected]>

LJH-LBJ force-pushed the Eagle-mulitmodal-support-Qwen2.5vl branch from e0fd906 to 8874e16 Compare August 14, 2025 12:32

Merge branch 'vllm-project:main' into Eagle-mulitmodal-support-Qwen2.5vl

f8af6b8

mergify bot added the needs-rebase label Aug 20, 2025

LJH-LBJ force-pushed the Eagle-mulitmodal-support-Qwen2.5vl branch from 828fc1e to f8af6b8 Compare August 21, 2025 11:27

Junhong added 6 commits August 21, 2025 19:28

fix bug

b242260

Signed-off-by: Junhong <[email protected]>

support M-RoPE in eagle

6b682de

Signed-off-by: Junhong <[email protected]>

fix bug for SupportsEagle3 and graph compile

475ce2b

Signed-off-by: Junhong <[email protected]>

fix bug

7df533c

Signed-off-by: Junhong <[email protected]>

Merge remote-tracking branch 'origin' into Eagle_support_mrope

fdb7cb2

fix bug

ce18a91

Signed-off-by: Junhong <[email protected]>

DarkLight1337 reviewed Sep 26, 2025

View reviewed changes

vllm/v1/spec_decode/eagle.py Outdated Show resolved Hide resolved

Merge branch 'main' into Eagle-mulitmodal-support-Qwen2.5vl

23bf946

Signed-off-by: Junhong Liu <[email protected]>

mergify bot removed the needs-rebase label Sep 26, 2025

opt

6d2882f

Signed-off-by: Junhong <[email protected]>

DarkLight1337 reviewed Sep 26, 2025

View reviewed changes

vllm/v1/spec_decode/eagle.py Outdated Show resolved Hide resolved

opt

cbe0744

Signed-off-by: Junhong <[email protected]>

DarkLight1337 reviewed Sep 26, 2025

View reviewed changes

vllm/v1/worker/gpu_model_runner.py Outdated Show resolved Hide resolved

opt

e5e29e0

Signed-off-by: Junhong <[email protected]>

DarkLight1337 approved these changes Sep 26, 2025

View reviewed changes

DarkLight1337 added the ready ONLY add when PR is ready to merge/full CI is needed label Sep 26, 2025

fix pre-commit

4e24437

Signed-off-by: Junhong <[email protected]>

fix bug

c38907b

Signed-off-by: Junhong <[email protected]>

mergify bot added the needs-rebase label Sep 27, 2025

Merge branch 'main' into Eagle-mulitmodal-support-Qwen2.5vl

6096054

Signed-off-by: Junhong Liu <[email protected]>

mergify bot removed the needs-rebase label Sep 27, 2025

DarkLight1337 enabled auto-merge (squash) September 27, 2025 02:26

DarkLight1337 merged commit f1d53d1 into vllm-project:main Sep 27, 2025
52 checks passed

benchislett mentioned this pull request Oct 8, 2025

[Spec-Decode] Support piecewise cudagraphs for Eagle head #25109

Merged

5 tasks

Uh oh!

[Multimodal][Speculative Decoding]Eagle Eagle3 mm support, enablement on qwen2.5vl #22872

[Multimodal][Speculative Decoding]Eagle Eagle3 mm support, enablement on qwen2.5vl #22872

Uh oh!

Conversation

david6666666 commented Aug 14, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

(Optional) Documentation Update

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

github-actions bot commented Aug 14, 2025

Uh oh!

DarkLight1337 Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

DarkLight1337 left a comment

Choose a reason for hiding this comment

Uh oh!

22quinn commented Aug 16, 2025

Uh oh!

mergify bot commented Aug 20, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DarkLight1337 left a comment

Choose a reason for hiding this comment

Uh oh!

DarkLight1337 commented Sep 27, 2025

Uh oh!

mergify bot commented Sep 27, 2025

Uh oh!

LJH-LBJ commented Sep 27, 2025

Uh oh!

Uh oh!

taoyun951753 commented Sep 29, 2025

Uh oh!

taoyun951753 commented Sep 29, 2025

Uh oh!

ggg-s commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DarkLight1337 commented Sep 29, 2025

Uh oh!

benchislett commented Oct 8, 2025

Uh oh!

taoyun951753 commented Oct 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

david6666666 commented Aug 14, 2025 •

edited by github-actions bot

Loading

ggg-s commented Sep 29, 2025 •

edited

Loading