[WIP][V1] Ray executor #10725
[WIP][V1] Ray executor #10725
Conversation
rkooo567
requested review from
WoosukKwon,
alexm-redhat,
comaniac,
njhill,
robertgshaw2-redhat and
ywang96
as code owners
|
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can do one of these:
🚀 |
| if use_message_queue_broadcaster and self.world_size > 1: | ||
| self.mq_broadcaster = MessageQueue.create_from_process_group( | ||
| self.cpu_group, 1 << 22, 6) | ||
| # if use_message_queue_broadcaster and self.world_size > 1: |
There was a problem hiding this comment.
I think this is not needed anymore for V1?
| @classmethod | ||
| def _get_executor_cls(cls, vllm_config: VllmConfig): | ||
| return GPUExecutor | ||
| if vllm_config.parallel_config.distributed_executor_backend == "ray": |
There was a problem hiding this comment.
We also need to have this in AsyncLLM?
There was a problem hiding this comment.
I remeber to hear AsyncLLM will be removed or something. Can you sync with them before supporting it?
|
This pull request has merge conflicts that must be resolved before it can be |
|
Closing as stale, feel free to re-open if you feel this PR is still relevant. I believe the Ray executor was addd in https://github.com/vllm-project/vllm/pull/13064/files#diff-48df0147dfa8eebb4792e6d88a0c9b42f69f7fe54409c41f76ca71bd2c76ba9c. |
3 participants
This PR supports Ray executor. We are only planning to support Ray compiled graph version backend for V1 which solves the performance problems raised before.
Detailed architecture is same as #9856 (which also matches the design in V0 with SPMD=True)
TODO