FusedMoE support for the Transformers backend

[hmellor]

MoE models do already work with the Transformers backend, but the performance is not ideal because FusedMoE is not used. This means that each expert gets a dedicated linear layer which is executed in a for loop on the Transformers side.

Depends on

• MoE + vllm = 😻  huggingface/transformers#40132 - so that FusedMoE.forward is called correctly in the Transformers modelling code
• Fix GPTQ model loading in Transformers backend #25770 - prerequisite for supporting GPTQ for MoE models

Changes

This PR adds:

• MoE model detection to ModelConfig._get_transformers_backend_cls based on the number of experts reported by the HF config
• FusedMoE.weight_loader handling to AutoWeightsLoader.load_weights which is only triggered if expert_mapping is passed to AutoWeightsLoader.load_weights (i.e. it's opt-in so won't break any existing custom weight loading)
• TransformersMoEBase which can be subclassed and adds the necessary logic to substitute in FusedMoE
  • Modules named experts and which are instances of torch.nn.ModuleList are replaced with FusedMoE modules
  • load_weights passes the expert_mapping directly to AutoWeightsLoader.load_weights
• TransformersMoEModel, TransformersMoEForCausalLM, TransformersMoEForMultimodalLM which leverages Python's MRO to add the MoE logic to their non-MoE counterparts

Performance

Before this PR the Transformers backend for MoE's was 24x slower than the vLLM implementation. This is largely due to the Transformers modelling code not being CUDA graphs compilable (--enforce-eager was needed) and a for loop iterating over the experts. This should be fixed by the PRs we depend on though.

As you can see from the results below the average performance of the Transformers backend is <3% worse than the dedicated vLLM implementation!

Serve commands:

# vLLM reference
vllm serve Qwen/Qwen3-30B-A3B --model-impl vllm
# Transformers backend
vllm serve Qwen/Qwen3-30B-A3B --model-impl transformers

Benchmark command:

vllm bench serve --model Qwen/Qwen3-30B-A3B --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 500 --request-rate inf --save-result

Results:

Metric	vllm	transformers	Diff (%)
Request Throughput (req/s)	20.0	19.5	-2.7%
Output Throughput (tok/s)	4397	4278	-2.7%
Total Token Throughput (tok/s)	8536	8305	-2.7%
Mean TTFT (ms)	1770	1794	+1.4%
P99 TTFT (ms)	3174	3214	+1.2%
Mean TPOT (ms)	63.6	65.2	+2.5%
P99 TPOT (ms)	204.5	207.1	+1.3%
Mean ITL (ms)	39.5	40.4	+2.3%

[gemini-code-assist]

Code Review

This pull request introduces support for Mixture-of-Experts (MoE) models within the Transformers backend. The changes are well-structured, including refactoring ModelConfig to handle MoE-specific properties, registering new MoE model classes, and adding a TransformersMoEBase class to manage the replacement of standard MoE layers with vLLM's FusedMoE. A critical issue was found where top_k and intermediate_size for the FusedMoE layer were hardcoded, which would cause issues for many MoE models. A detailed comment with a suggested fix has been provided to address this.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[ArthurZucker]

looks nice already!

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @hmellor.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[ProExpertProg]

Not very familiar with the backend but was curious, just some thoughts

[hmellor]

Not very familiar with the backend but was curious, just some thoughts

Thanks for the interest! Slowly but surely I'm bolting on more and more functionality to it 🚀