[Core] Use CpuGpuBuffer for block table tensors
#24795
Conversation
Signed-off-by: Nick Hill <[email protected]>
njhill
requested review from
WoosukKwon,
alexm-redhat,
comaniac,
robertgshaw2-redhat and
ywang96
as code owners
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request refactors the BlockTable class to use the CpuGpuBuffer helper for managing tensors that exist on both CPU and GPU, specifically for block_table and slot_mapping. This change simplifies the code by centralizing the creation and management of these paired buffers, removing redundant manual handling of CPU, GPU, and NumPy tensor versions. The changes are well-contained within vllm/v1/worker/block_table.py, and the necessary adjustments in vllm/v1/worker/gpu_model_runner.py to adapt to the updated BlockTable API have been correctly applied. The refactoring improves code clarity and maintainability without introducing any apparent issues.
DarkLight1337
left a comment
DarkLight1337
left a comment
There was a problem hiding this comment.
cc @WoosukKwon please confirm if this is ok to avoid conflicting with your refactoring efforts
WoosukKwon
left a comment
WoosukKwon
left a comment
There was a problem hiding this comment.
My PR will rewrite the block table entirely, but this change looks good for now.
Signed-off-by: Nick Hill <[email protected]>
Pull Request is not mergeable
Signed-off-by: Nick Hill <[email protected]>
|
The failing test is also failing on main :( example: https://buildkite.com/vllm/ci/builds/30944#01995329-1ec7-4d17-abdf-0283fa8115f5 |
vllm-project/vllm#24795 and vllm-project/vllm#24615 and vllm-project/vllm#24078 --------- Signed-off-by: Agata Dobrzyniewicz <[email protected]>
vllm-project/vllm#24795 and vllm-project/vllm#24615 and vllm-project/vllm#24078 --------- Signed-off-by: Agata Dobrzyniewicz <[email protected]> Signed-off-by: slokesha <[email protected]>
Signed-off-by: Nick Hill <[email protected]>
Signed-off-by: Nick Hill <[email protected]> Signed-off-by: charlifu <[email protected]>
Signed-off-by: Nick Hill <[email protected]> Signed-off-by: xuebwang-amd <[email protected]>
Signed-off-by: Nick Hill <[email protected]>
Signed-off-by: Nick Hill <[email protected]> Signed-off-by: xuebwang-amd <[email protected]>
4 participants
In the gpu model runner input batch.