[V1] Support interleaved modality items

[ywang96]

Partially unblock #15130 - the current workaround can be inefficient in case there are a large number of interleaved items in the current batch (e.g, <image><video><image><video><image>), but this should be a rare case as typically these embeddings are individually fairly big.

Note use_audio_in_video is not yet covered by this PR and will require a bit of more work, since it will require mixing of modality items/interleaved embeddings.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[DarkLight1337]

We still need to update get_multimodal_embeddings to support interleaved modality inputs e2e, am I correct?

[ywang96]

We still need to update get_multimodal_embeddings to support interleaved modality inputs e2e, am I correct?

No I don't think we need to - this is because model runner will try to batch only consecutive inputs of the same modality, or otherwise execute encoder individually.

vllm/vllm/v1/worker/gpu_model_runner.py

Lines 839 to 865 in 3f532cb

	# Batch mm inputs as much as we can: if a request in the batch has
	# multiple modalities or a different modality than the previous one,
	# we process it separately to preserve item order.
	# FIXME(ywang96): This is a hacky way to deal with multiple modalities
	# in the same batch while still being able to benefit from batching
	# multimodal inputs. The proper solution should be reordering the
	# encoder outputs.
	grouped_mm_inputs_list = group_mm_inputs_by_modality(mm_inputs)
	encoder_outputs = []
	for grouped_mm_inputs in grouped_mm_inputs_list:
	batched_mm_inputs = MultiModalKwargs.batch(grouped_mm_inputs)
	batched_mm_inputs = MultiModalKwargs.as_kwargs(batched_mm_inputs,
	device=self.device)
	# Run the encoder.
	# `curr_group_outputs` is either of the following:
	# 1. A tensor of shape (num_items, feature_size, hidden_size)
	# in case feature_size is fixed across all multimodal items.
	# 2. A list or tuple (length: num_items) of tensors, each of shape
	# (feature_size, hidden_size) in case the feature size is dynamic
	# depending on the input multimodal items.
	curr_group_outputs = self.model.get_multimodal_embeddings(
	**batched_mm_inputs)
	for output in curr_group_outputs:
	encoder_outputs.append(output)

[DarkLight1337]

Hmm in that case, can we add a test to verify this e2e? We can use llava-hf/llava-onevision-qwen2-0.5b-ov-hf which lets us fit interleaved image and videos in memory even in CI environment.

[ywang96]

Hmm in that case, can we add a test to verify this e2e? We can use llava-hf/llava-onevision-qwen2-0.5b-ov-hf which lets us fit interleaved image and videos in memory even in CI environment.

Yea we can - I'll add a test for this during the day

[DarkLight1337]

Let's also update the V1 User Guide to remove the part about interleaved modality not supported.

[ywang96]

llava-hf/llava-onevision-qwen2-0.5b-ov-hf

Interestingly - this model seems to ignore the order of modalities from the frontend and the processor will always group modalities together

[DarkLight1337]

llava-hf/llava-onevision-qwen2-0.5b-ov-hf

Interestingly - this model seems to ignore the order of modalities from the frontend and the processor will always group modalities together

By "frontend" are you referring to online inference? I think this is because the chat template is detected as "string" format which disables interleaved inputs.

[DarkLight1337]

There appears to be an issue with the test runners for validating mixed-modality inputs.

As discussed offline, let's merge this PR first to unblock Qwen2.5-Omni, and work on fixing this in another PR.

[ywang96]

Add a simple test to verify interleaving modalities generates a different result in greedy decoding.

[ywang96]

Test model runner previously didn't support testing with multiple modalities and this PR fixed it.