[Feature]: Streaming multi-modal input/output

[DarkLight1337]

This is a tracking issue for enabling streaming MM I/O.

Outline

Streaming input:

• Support streaming multi-modal inputs at API level.
• Handle streaming inputs in the multi-modal processor.
• Define an interface for models to indicate support for streaming inputs.
• Update V1 model runner and scheduler to handle partial MM encoding requests (this is the hardest part IMO)

Streaming output:

• Implement RequestOutputKind.DELTA for multi-modal outputs in V1 output processor.
• Support streaming multi-modal outputs at API level.

Notes

• Currently we are waiting for AWS's proposal
• Take inspiration from [Model][VLM] Add Qwen2.5-Omni model support (end-to-end full support) #16347?
• See also #22695

[qandrew]

Hi! some quick questions

• is the preferred way to implement MM streaming via responsesAPI or the legacy chat competions API?

[DarkLight1337]

I think we can enable it for both, the API additions should be easy compared to what needs to be changed in the core engine.

[TKONIY]

Hi @DarkLight1337, I am interested in helping implemented the MM I/O support, e,g, supporting models like Qwen3-Omni. Before that I have contributed to the first video support in vLLM. How could start to find an issue to contribute to?

cc @ywang96