[Bugfix] Merge MM embeddings by index instead of token IDs

[DarkLight1337]

This PR fixes a mismatch in merging multi-modal embeddings when the model itself generates embedding placeholder tokens such as <image>. Although this error mainly occurs in V1, it can possibly occur in V0 as well. This PR focuses on the V1 case.

For V0 users, you can work around this by setting top_p so that the model has no chance of generating such tokens.

FIX #15677
FIX #15764
FIX #23891
FIX #23954
FIX #24456

Breaking change for model developers

This PR has updated SupportsMultiModal.get_input_embeddings to support passing is_multimodal mask and added a default implementation so that there is no need to override it in most cases. OOT/WIP models should either remove their override to use the default implementation, or update their override to accept is_multimodal and do_language_embed_multimodal arguments.

Text-only model developers should ensure that their models have implemented get_input_embeddings to continue using them in vLLM.

Breaking change for model runner plugins

In order to continue supporting multimodal models, you should update _gather_mm_embeddings method to build up and return the is_mm_embed mask, then pass it to the model.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[NickLucche]

Fixed the tpu issues, thanks @DarkLight1337 !

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @DarkLight1337.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @DarkLight1337.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[DarkLight1337]

We will not include this in the v0.11 release because of the breaking change. But it should be fine to merge this into main branch since the release branch has been cut already.