Skip to content

[Core] Efficient transmission for CPU prefix caching, based on PR#10874 #11099

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 36 commits into
base: main
Choose a base branch
from
Open

[Core] Efficient transmission for CPU prefix caching, based on PR#10874 #11099

wants to merge 36 commits into from

Conversation

lixiaobai09
Copy link

@lixiaobai09 lixiaobai09 commented Dec 11, 2024 •
edited by github-actions bot
Loading

We implement an efficient CPU prefix caching, including:

  1. Based on an efficient CPU KV Block Cache Manager (PR#10874).
  2. A data transmission optimization to overlap layer-wise block swapping and forward computing.
  3. A data transmission optimization to overlap request-level block swapping and computing by delaying the transmission requests one step.

ApostaC and others added 13 commits December 3, 2024 20:59
@ApostaC @KuntaiDu
Move to a new branch to fix the DCO issues.
f60a8fa 
Signed-off-by: ApostaC <[email protected]>
Co-authored-by: KuntaiDu <[email protected]>
@ApostaC
[Fix] the failed unit tests
e6654f2 
Signed-off-by: ApostaC <[email protected]>
@ApostaC
[Fix] CPU offloading not working bug and [fix] unit test and format i…
ba6c9e3 
…ssues

Signed-off-by: ApostaC <[email protected]>
@ApostaC
[fix] broken tests for cpu offloading allocator
1c94985 
Signed-off-by: ApostaC <[email protected]>
@ApostaC
[Fix] add the call to get_physical_block_ids
daab0d6 
Signed-off-by: ApostaC <[email protected]>
@ApostaC
[Add] faster unsafe implementation for get_physical_block_id
919e5e3 
Signed-off-by: ApostaC <[email protected]>
Feat: support CSR format to construct the swapped blocks
0638211 
sequence IDs with each swapped blocks

Signed-off-by: Dahai Tang <[email protected]>
@ApostaC
Merge branch 'main' into yihua-cpu-offloading2
52185bf
@ApostaC
Updating the benchmark script with correct usage instructions
505e60c 
Signed-off-by: ApostaC <[email protected]>
@ApostaC
make yapf happy
a517a29 
Signed-off-by: ApostaC <[email protected]>
@ApostaC
fix format checker issues
789b00e 
Signed-off-by: ApostaC <[email protected]>
Feat: layer-wise transmission
6d5841f 
Signed-off-by: Dahai Tang <[email protected]>
Merge branch 'yihua-cpu-offloading2' of https://github.com/KuntaiDu/vllm
5927066 
 into cpu-offloading2

Signed-off-by: Dahai Tang <[email protected]>
@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

  • Add ready label to the PR
  • Enable auto-merge.

🚀

@mergify mergify bot added the frontend label Dec 11, 2024
@mergify Mergify
Copy link

mergify bot commented Dec 11, 2024

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @lixiaobai09.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Dec 11, 2024
@mergify mergify bot removed the needs-rebase label Dec 11, 2024
Dahai Tang and others added 15 commits December 12, 2024 04:27
Fix: lint checker
47c3557 
Signed-off-by: Dahai Tang <[email protected]>
Fix: cpu offloading block allocator tester
894ab90 
Signed-off-by: Dahai Tang <[email protected]>
Fix: get_cache_engine while self.cache_engine is None
80c8c4e 
Signed-off-by: Dahai Tang <[email protected]>
Fix: unit tester
206bb97 
Signed-off-by: Dahai Tang <[email protected]>
@ApostaC
Merge remote-tracking branch 'upstream/main' into yihua-cpu-offloading2
a0b5061
@ApostaC
Fix the compatibility witht the latest main
2648fa5 
Signed-off-by: ApostaC <[email protected]>
@ApostaC
Fix the typo uncacherd -> uncached
278e166 
Signed-off-by: ApostaC <[email protected]>
Fix: split swap taks of preempting and offloading
3a3075e 
Signed-off-by: Dahai Tang <[email protected]>
Merge branch 'yihua-cpu-offloading2' of https://github.com/KuntaiDu/vllm
833623f 
 into cpu-offloading2
Merge branch 'main' of https://github.com/vllm-project/vllm into cpu-…
c3a8ae8 
…offloading2

Signed-off-by: Dahai Tang <[email protected]>
Fix: exclude cache_engine argument for input dump
0f5e9e3 
Signed-off-by: Dahai Tang <[email protected]>
Merge branch 'main' of https://github.com/vllm-project/vllm into cpu-…
c227564 
…offloading2

Signed-off-by: Dahai Tang <[email protected]>
Merge branch 'main' of https://github.com/vllm-project/vllm into cpu-…
3ba57a2 
…offloading2
Perf: aviod call get_and_reset_swap in every decode step
936248a 
Signed-off-by: Dahai Tang <[email protected]>
Feat: overlap transmission in scheduler
ae0f73c 
Signed-off-by: Dahai Tang <[email protected]>
@mergify Mergify
Copy link

mergify bot commented Dec 18, 2024

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @lixiaobai09.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Dec 18, 2024
Dahai Tang added 2 commits December 18, 2024 09:30
@mergify mergify bot removed the needs-rebase label Dec 18, 2024
Dahai Tang added 2 commits December 18, 2024 12:35
Fix: test_scheduler unit test
5358aee 
Signed-off-by: Dahai Tang <[email protected]>
Minor: test_cpu_offloading
673869f 
Signed-off-by: Dahai Tang <[email protected]>
@mergify Mergify
Copy link

mergify bot commented Feb 5, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @lixiaobai09.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Feb 5, 2025
@mergify mergify bot added the tpu Related to Google TPUs label Mar 27, 2025
@mergify mergify bot added the performance Performance-related issues label Jun 23, 2025
@github-actions GitHub Actions
Copy link

This pull request has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this pull request should remain open. Thank you!

@github-actions github-actions bot added the stale Over 90 days of inactivity label Sep 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth is a code owner

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@njhill njhill Awaiting requested review from njhill njhill is a code owner

@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 is a code owner

@comaniac comaniac Awaiting requested review from comaniac comaniac is a code owner

@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat alexm-redhat is a code owner

@zhuohan123 zhuohan123 Awaiting requested review from zhuohan123 zhuohan123 is a code owner

@youkaichao youkaichao Awaiting requested review from youkaichao youkaichao is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

frontend needs-rebase performance Performance-related issues speculative-decoding stale Over 90 days of inactivity tpu Related to Google TPUs v1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@lixiaobai09 @ApostaC