[UX] Replace VLLM_ALL2ALL_BACKEND with --all2all-backend #26732
Conversation
Signed-off-by: mgoin <[email protected]>
mgoin
requested review from
ProExpertProg,
WoosukKwon,
alexm-redhat,
comaniac,
hmellor,
houseroad,
njhill,
robertgshaw2-redhat,
simon-mo,
tlrmchlsmth,
yewentao256,
youkaichao and
ywang96
as code owners
|
Documentation preview: https://vllm--26732.org.readthedocs.build/en/26732/ |
Signed-off-by: mgoin <[email protected]>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new command-line argument --all2all-backend to replace the VLLM_ALL2ALL_BACKEND environment variable, which is now deprecated. The changes span across documentation, configuration files, and various model execution layers to adopt the new configuration method. This is a good UX improvement. My review includes one high-severity comment regarding a potential logic mismatch between the implementation and the stated behavior in the pull request description concerning the precedence of the new argument versus the old environment variable.
Signed-off-by: mgoin <[email protected]>
yewentao256
left a comment
yewentao256
left a comment
There was a problem hiding this comment.
LGTM, thanks for the work!
…t#26732) Signed-off-by: mgoin <[email protected]> Signed-off-by: 1994 <[email protected]>
…t#26732) Signed-off-by: mgoin <[email protected]> Signed-off-by: Dhruvil Bhatt <[email protected]>
…t#26732) Signed-off-by: mgoin <[email protected]> Signed-off-by: bbartels <[email protected]>
…t#26732) Signed-off-by: mgoin <[email protected]>
…t#26732) Signed-off-by: mgoin <[email protected]>
…t#26732) Signed-off-by: mgoin <[email protected]> Signed-off-by: xuebwang-amd <[email protected]>
…t#26732) Signed-off-by: mgoin <[email protected]> Signed-off-by: xuebwang-amd <[email protected]>
…t#26732) Signed-off-by: mgoin <[email protected]> Signed-off-by: 0xrushi <[email protected]>
…t#26732) Signed-off-by: mgoin <[email protected]> Signed-off-by: 0xrushi <[email protected]>
…t#26732) Signed-off-by: mgoin <[email protected]>
…t#26732) Signed-off-by: mgoin <[email protected]>
…t#26732) Signed-off-by: mgoin <[email protected]>
5 participants
Purpose
We have solidified the use of various all-to-all backends for the moe communication during expert parallelism. It is past time to graduate the environment variable to a proper config.
Old way (still works but deprecated):
VLLM_ALL2ALL_BACKEND=deepep_high_throughput vllm serve openai/gpt-oss-20b --data-parallel-size 2 --enable-expert-parallelNew way:
vllm serve openai/gpt-oss-20b --data-parallel-size 2 --enable-expert-parallel --all2all-backend deepep_high_throughputThe env var will still work by itself, but it will be overriden by the argument if both are set. Users will see a deprecation warning encouraging them to use the argument instead.
Test Plan
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.