【Spec Decode】support async scheduling with eagle speculative decoding

[woodlgz]

Previous work 19970 and 23569 solved async scheduling with common scenarios leaving speculative decoding not yet suppoted.

Purpose

this pull request targets to make eagle speculative decoding work with async-scheduling.

Test Plan

we conducted benchmark with following command under nvidia L20 machine:

benchmark server

VLLM_ATTENTION_BACKEND=FLASH_ATTN vllm serve meta-llama/Llama-3.1-8B-Instruct --speculative-config '{"method": "eagle3", "model": "yuhuili/EAGLE3-LLaMA3.1-Instruct-8B", "num_speculative_tokens": 4, "disable_padded_drafter_batch": false}' --max-model-len 2048 --no-enable-prefix-caching --async-scheduling

benchmark client

vllm bench serve --port 8000 --save-result --backend vllm --model meta-llama/Llama-3.1-8B-Instruct --endpoint /v1/completions --dataset-name hf --dataset-path philschmid/mt-bench --num-prompts 80 --max-concurrency 1

Test Result

version	mean TPOT(ms)	P99 TPOT(ms)	Mean ITL(ms)	P99 ITL(ms)
main	9.91	14.94	28.14	28.52
this pr	9.57	14.42	27.17	28.15

---

mean TPOT/ITL peformance gain up to 3.4%.
step gaps before optimization:

step gaps after optimization:

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

[gemini-code-assist]

Code Review

This pull request introduces support for asynchronous scheduling with Eagle speculative decoding. The changes are extensive, touching the scheduler, model runner, and speculative decoding logic. The core of the implementation is to make the speculative decoding flow asynchronous to improve performance, primarily by avoiding CPU-GPU synchronizations. This is achieved by using GPU events for synchronization, caching intermediate GPU tensors between steps, and introducing a 'fix-up' step to correct token counts based on rejected speculative tokens from the previous step. The changes appear to be well-thought-out and correctly implement the asynchronous logic. I have reviewed the code and did not find any issues of high or critical severity.

[lhtin]

Is it duplicate of this PR(#24799)?

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @woodlgz.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork