[Bugfix] Fix glm4.1v video inference issue

[Isotr0py]

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

• Fix [Bug]: GLM-4.1V-Thinking ValueError #21811 and Fix [Bug]: run glm4.1v ,ValueError: Attempted to assign 100 = 100 multimodal tokens to 30000 placeholders #21516
• Transformers changed GLM4.1V's video_grid_thw to fix batch inference after v4.53.2 (fix Glm4v batch videos forward huggingface/transformers#39172), which broke the video support GLM-4.1V in multimodal processor.

video_grid_thw before:

tensor([[ 1, 26, 46],
        [ 1, 26, 46],
        [ 1, 26, 46],
        [ 1, 26, 46],
        [ 1, 26, 46]])

video_grid_thw currently:

tensor([[ 5, 26, 46]])

TODO

• Add regression test

Test Plan

Test Result

(Optional) Documentation Update

[gemini-code-assist]

Code Review

This pull request aims to fix a video inference issue in GLM-4.1V caused by a change in the transformers library. While the change correctly adapts to the new video_grid_thw format for a single video, it introduces a critical bug in handling multiple videos within a single request. The proposed change incorrectly uses data from only the last video in a batch, ignoring the others. My review includes a fix for this issue.

[gemini-code-assist]

This change introduces a bug when processing multiple videos in a single request. The video_outputs variable is updated in each iteration of the loop over videos. By using video_outputs["video_grid_thw"] after the loop, you are only using the video_grid_thw from the last video processed.

The video_grid_thw_lst correctly accumulates the grid information for all videos. You should concatenate the tensors in this list to form the final batched video_grid_thw, as was done previously.

This is a critical issue that will lead to incorrect outputs for multi-video inputs.

Suggested change

	video_outputs = dict(
	pixel_values_videos=torch.cat(pixel_values_videos_lst),
	video_grid_thw=torch.cat(video_grid_thw_lst),
	video_grid_thw=video_outputs["video_grid_thw"],
	)
	video_outputs = dict(
	pixel_values_videos=torch.cat(pixel_values_videos_lst),
	video_grid_thw=torch.cat(video_grid_thw_lst),
	)

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[DarkLight1337]

Thanks for fixing!

[DarkLight1337]

Can you merge from main?