[RFC]: To Inductor partition or to not Inductor partition (by default in v0.11.1)

[ProExpertProg]

tl;dr: Inductor partition is a semi-experimental feature in torch 2.9 that increases performance but also compile time and has risks, we should decide whether to enable it by default or not in 0.11.1.

Motivation.

Currently, on vLLM main, piecewise cudagraphs are achieved using Dynamo partitioning (splitting the fx.Graph before it enters Inductor). That makes piecewise cudagraphs incompatible with compilation passes that need to see the whole graph to work - like attention+quant fusion and sequence parallelism (and hence async tp). Apart from allreduce+rmsnorm(+quant) fusion, those are the passes that bring the most benefit.

The vLLM x torch.compile collaboration yielded a custom Inductor partitioning solution (RFC: #23261, PR: #24281). It gives better performance by itself by reducing cudagraph replay overhead and it allows piecewise cudagraphs to be compatible with fullgraph custom passes. It requires torch==2.9, with a couple of monkeypatches and workarounds (listed below) in vLLM but they are not too bad. It also significantly increases cold-start (first-time) compilation time. Finally, it is somewhat of an experimental feature, and we've resolved most of the known issues (so far), but there could be more.

Monkeypatches & workarounds:

• [BugFix] Patch inductor partitioning logic #26735: monkeypatch for partition rules
• [BugFix] Patch inductor memory plan logic #26878: monkeypatch for memory plan output naming
• [BugFix] Work around graph partition x torch.compile cache issue #26956: AOT caching issue workaround
• disable graph partition in custom op #26952: nested torch.compiles need to disable graph partitioning (not an issue, just might need to be applied to more places)

Current known issues:

• [Bug]: Another issue with Inductor partition codegen for attn+nvfp4 quant fusion #26988: attention-nvfp4 fusion + inductor partition causes Inductor codegen error
• a few more CI issues: [DO NOT MERGE] 2.9, Inductor partition, standalone compile, monkeypatch fix(es) #26738, hoping to resolve these by EOD Friday

Proposed Change.

The big question to resolve is whether to enable Inductor partition by default in 0.11.1 or not. The upside is increased performance, the downside is risk of breakage and increased compilation time. Inductor partitioning has gone through a decent amount of manual testing as well as vLLM CI (some issues are still being resolved). The performance benefit is between 2-10% for various models and QPS regimes. The startup cost is around 2-5x depending on the model - although warm start is actually slightly faster as there are fewer artifacts to load. More information on performance is below.

If we choose to enable Inductor partition by default, there's also a second question of whether to enable attention+quant fusion by default (up to additional 2-6%) and SP+AsyncTP by default (numbers TBD). Both of them require Inductor partition (or they have to disable piecewise compilation and cudagraphs). A downside would be that we haven't benchmarked these extensively across different kinds of hardware. For example, attention+quant fusion causes a slowdown on llama-70B tp=4 on Blackwell with torch==2.9 (cause unknown). So we're likely not going to enable these by default.

Note that even if enabled by default, that users can always disable Inductor partitioning. With the planned addition of optimization levels (RFC: #20283, PR: #26847, more below), it will be as easy as -O1. This way there's an easy way to control the tradeoff between performance and startup (which will only grow over time), especially useful for development.

What we want to avoid is souring users by increasing startup time while not delivering as much speedup as initially planned (the original plan for this release was all fusion passes improved and on by default in -O2). However, vLLM is known for being at the forefront of performance and innovation and we can always hotfix issues if necessary. And the earlier we turn it on, the earlier we can find any remaining issues, improve the feature, and deliver better performance to users.

We could also make -O1 (with FULL_AND_PIECEWISE cudagraphs) the default for now and let users opt into inductor partition and compile passes with -O2. In the next release, we can change the default from -O1 to -O2. In fact this would be my preferred approach if we don't want inductor partition by default.

Feedback Period.

By the time we release 0.11.1 - late Friday 10/17 or early weekend.

CC List.

@simon-mo @WoosukKwon @youkaichao @zou3519 @BoyuanFeng @angelayi @pavanimajety @tlrmchlsmth @robertgshaw2-redhat @mgoin @alexm-redhat @morrison-turnansky

Any Other Things.

Appendix: Optimization levels

• All optimization levels are achievable manually via compilation of flags
• All flags can be overridden

End goal:

• -O0: no optimization/compilation, fast startup (basically --enforce-eager)
• -O1: fast compilation: Dynamo partition, PIECEWISE cudagraphs, simple compile passes (SiluMul+quant fusion)
• -O2: full optimization (default): Inductor partition, FULL_AND_PIECEWISE cudagraphs, all compile passes
• -O3: max autotune (future): O2 with additional compile sizes and autotuning

Appendix: Benchmarking results

These numbers all use 2.9 + #24604. The H100 SP-AsyncTP numbers also use #26975.

On B200, the following settings were used:

• common: --kv-cache-dtype=fp8 --no-enable-prefix-caching -O.pass_config.enable_noop=true
• dynamo partition: -O.use_inductor_graph_partition=False -O.cudagraph_mode=FULL_AND_PIECEWISE
• inductor partition: -O.use_inductor_graph_partition=True -O.cudagraph_mode=FULL_AND_PIECEWISE
• no partition: -O.use_inductor_graph_partition=False -O.cudagraph_mode=FULL_DECODE_ONLY -O.splitting_ops=[]
• fusion: -O.pass_config.enable_attn_fusion=true
• 2.8: means torch==2.8 was used

1xB200, redhatai/meta-llama-3.1-8B-Instruct-FP8

4xB200, redhatai/meta-llama-3.1-70B-Instruct-FP8

4xB200, nvidia/Llama-4-Scout-17B-16E-Instruct-FP8

---

On H100, the following additional settings were used:

• common: --gpu_memory_utilization=0.8
• async_tp: -O.pass_config.enable_async_tp=true (implies -O.pass_config.enable_sequence_parallelism=true)

4xH100, redhatai/meta-llama-3.1-70B-Instruct-FP8

Startup Time

Taken from #24281, anecdotally I've seen larger models compile for up to 100s on cold start as well but that was the most I've seen.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[tomasruizt]

The startup cost is around 2-5x depending on the model

Does that mean that if startup is 1min now it can be 5min in the future? I've noticed that debugging models without --enforce-eager has very long cycle times because of the startup time. I'm afraid waiting 5min to hit a breakpoint would wreck productivity. The 1min we have today is already a significant hidden drag on productivity.

[ProExpertProg]

What model and deployment config are you seeing 1min on? This won't increase other parts of vllm startup, just Inductor compilation, and not more than what -O.splitting_ops=[] does today. We also plan on reducing this startup time overhead further in the future, this is just the current trade-off for this specific release.

How do you feel about being able to do -O1 to get a faster startup time?

[tomasruizt]

I get total startup times of 58s on Qwen/Qwen3-4B on an RTX3090.

code

import time
from vllm import LLM

if __name__ == "__main__":
    start_time = time.time()
    llm = LLM("Qwen/Qwen3-4B", enforce_eager=False)
    end_time = time.time()
    print(f"Time taken: {end_time - start_time} seconds")

logs.log

Edit: I haven't tried setting granular levels with -O. I will try this out during development.

[mgoin]

I don't think we should turn on this feature by default for 0.11.1.
We already have potential issues with performance/correctness with the upgrade to torch 2.9.0, so I'd prefer to take that as the "drastic change" this release since we don't have time to let this change sit on main several days before cutting the release.
Thinking about user optics, I don't want users to equate upgrading to torch 2.9.0 as vLLM takes longer to start. Maybe we can figure out ways to improve the cold start issues before enabling this by default with more time.

[tlrmchlsmth]

+1 to what @mgoin says. I'd also like to understand the cold start time implications across a wider set of models before making it a default.

[ProExpertProg]

@tlrmchlsmth we can add Deepseek, Qwen, GPT-OSS to the mix, any other models you want to see? We can also do llama-405B