Skip to content

[VLM] Support HF format Phi-4-MM model #17121

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 48 commits into from
Jul 27, 2025
Merged

[VLM] Support HF format Phi-4-MM model #17121

merged 48 commits into from
Jul 27, 2025

Conversation

Isotr0py
Copy link
Member

@Isotr0py Isotr0py commented Apr 24, 2025 •
edited by github-actions bot
Loading

  • Transformers has supported Phi4-MM natively, without scipy requirement. (Add Phi4 multimodal huggingface/transformers#36939)
  • Add HF-format Phi-4-MM support, so that we can remove scipy from dependency in the future after deprecating legacy phi4mm implementation.

Isotr0py added 6 commits April 24, 2025 22:15
@Isotr0py
init
203fbb4 
Signed-off-by: Isotr0py <[email protected]>
@Isotr0py
init
3a41833 
Signed-off-by: Isotr0py <[email protected]>
@Isotr0py
init example
4486373 
Signed-off-by: Isotr0py <[email protected]>
@Isotr0py
init example
d1a0a69 
Signed-off-by: Isotr0py <[email protected]>
@Isotr0py
fix projection
352b747 
Signed-off-by: Isotr0py <[email protected]>
@Isotr0py
fix model registry
22b2611 
Signed-off-by: Isotr0py <[email protected]>
@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@mergify mergify bot added the documentation Improvements or additions to documentation label Apr 24, 2025
@Isotr0py Isotr0py mentioned this pull request Apr 24, 2025
Closed
Isotr0py added 21 commits April 25, 2025 01:03
@Isotr0py
fix model loading
c3fe338 
Signed-off-by: Isotr0py <[email protected]>
@Isotr0py
clean up vit initialization
33d13ca 
Signed-off-by: Isotr0py <[email protected]>
@Isotr0py
fix embed names and processing info
7eac283 
Signed-off-by: Isotr0py <[email protected]>
@Isotr0py
fix audio processing info
4b65be4 
Signed-off-by: Isotr0py <[email protected]>
@Isotr0py
fix vision inference
4414a21 
Signed-off-by: Isotr0py <[email protected]>
@Isotr0py
fix vision inference
940e04f 
Signed-off-by: Isotr0py <[email protected]>
@Isotr0py
fix lora
d18f0b4 
Signed-off-by: Isotr0py <[email protected]>
@Isotr0py
add multi-image example
89f7545 
Signed-off-by: Isotr0py <[email protected]>
@Isotr0py
fix multi-image example
dbf6324 
Signed-off-by: Isotr0py <[email protected]>
@Isotr0py
fix audio
f77803f 
Signed-off-by: Isotr0py <[email protected]>
@Isotr0py
fix audio inference
a8c4547 
Signed-off-by: Isotr0py <[email protected]>
@Isotr0py
fix audio mlp
2b1fb2a 
Signed-off-by: Isotr0py <[email protected]>
@Isotr0py
clean up
9d15ac4 
Signed-off-by: Isotr0py <[email protected]>
@Isotr0py
fix audio tp
e246fea 
Signed-off-by: Isotr0py <[email protected]>
@Isotr0py
Merge branch 'vllm-project:main' into phi4-mm-hf
1dd185c
@Isotr0py
fix v1 inference
6416e26 
Signed-off-by: Isotr0py <[email protected]>
@Isotr0py
Merge branch 'vllm-project:main' into phi4-mm-hf
709e257
@Isotr0py
Merge branch 'vllm-project:main' into phi4-mm-hf
d0e6a08
@Isotr0py
update example and fix tp
2ee04be 
Signed-off-by: Isotr0py <[email protected]>
@Isotr0py
Merge branch 'vllm-project:main' into phi4-mm-hf
87bc8c7
@Isotr0py
add test
09d3990 
Signed-off-by: Isotr0py <[email protected]>
@Isotr0py
make pre-commit happy
04e238f 
Signed-off-by: Isotr0py <[email protected]>
@Isotr0py Isotr0py marked this pull request as ready for review July 14, 2025 15:53
@mergify Mergify
Copy link

mergify bot commented Jul 16, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Isotr0py.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Jul 16, 2025
@mergify mergify bot removed the needs-rebase label Jul 16, 2025
hmellor
hmellor reviewed Jul 18, 2025
View reviewed changes
@Isotr0py
address comments
dce8cb1 
Signed-off-by: Isotr0py <[email protected]>
hmellor
hmellor approved these changes Jul 23, 2025
View reviewed changes
Copy link
Member

@hmellor hmellor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM as long as tests pass

@DarkLight1337 DarkLight1337 added this to the v0.10.0 milestone Jul 23, 2025
@Isotr0py Isotr0py enabled auto-merge (squash) July 23, 2025 11:30
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 23, 2025
@vllm-bot vllm-bot merged commit eed2f46 into vllm-project:main Jul 27, 2025
79 of 81 checks passed
@Isotr0py Isotr0py deleted the phi4-mm-hf branch July 27, 2025 03:21
liuyumoye pushed a commit to liuyumoye/vllm that referenced this pull request Jul 31, 2025
@Isotr0py
[VLM] Support HF format Phi-4-MM model (vllm-project#17121)
bfdaf6a 
Signed-off-by: Isotr0py <[email protected]>
HsChen-sys pushed a commit to HsChen-sys/vllm that referenced this pull request Aug 1, 2025
@Isotr0py
[VLM] Support HF format Phi-4-MM model (vllm-project#17121)
873d9d4 
Signed-off-by: Isotr0py <[email protected]>
x22x22 pushed a commit to x22x22/vllm that referenced this pull request Aug 5, 2025
@Isotr0py @x22x22
[VLM] Support HF format Phi-4-MM model (vllm-project#17121)
186db1b 
Signed-off-by: Isotr0py <[email protected]>
Signed-off-by: x22x22 <[email protected]>
Pradyun92 pushed a commit to Pradyun92/vllm that referenced this pull request Aug 6, 2025
@Isotr0py
[VLM] Support HF format Phi-4-MM model (vllm-project#17121)
abf532b 
Signed-off-by: Isotr0py <[email protected]>
npanpaliya pushed a commit to odh-on-pz/vllm-upstream that referenced this pull request Aug 6, 2025
@Isotr0py @npanpaliya
[VLM] Support HF format Phi-4-MM model (vllm-project#17121)
705c09a 
Signed-off-by: Isotr0py <[email protected]>
jinzhen-lin pushed a commit to jinzhen-lin/vllm that referenced this pull request Aug 9, 2025
@Isotr0py @jinzhen-lin
[VLM] Support HF format Phi-4-MM model (vllm-project#17121)
3d4ae50 
Signed-off-by: Isotr0py <[email protected]>
Signed-off-by: Jinzhen Lin <[email protected]>
paulpak58 pushed a commit to paulpak58/vllm that referenced this pull request Aug 13, 2025
@Isotr0py @paulpak58
[VLM] Support HF format Phi-4-MM model (vllm-project#17121)
5be3cd0 
Signed-off-by: Isotr0py <[email protected]>
Signed-off-by: Paul Pak <[email protected]>
diegocastanibm pushed a commit to diegocastanibm/vllm that referenced this pull request Aug 15, 2025
@Isotr0py @diegocastanibm
[VLM] Support HF format Phi-4-MM model (vllm-project#17121)
15d9206 
Signed-off-by: Isotr0py <[email protected]>
Signed-off-by: Diego-Castan <[email protected]>
epwalsh pushed a commit to epwalsh/vllm that referenced this pull request Aug 28, 2025
@Isotr0py @epwalsh
[VLM] Support HF format Phi-4-MM model (vllm-project#17121)
4062a17 
Signed-off-by: Isotr0py <[email protected]>
zhewenl pushed a commit to zhewenl/vllm that referenced this pull request Aug 28, 2025
@Isotr0py @zhewenl
[VLM] Support HF format Phi-4-MM model (vllm-project#17121)
b6e3a36 
Signed-off-by: Isotr0py <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DarkLight1337 DarkLight1337 DarkLight1337 left review comments

@hmellor hmellor hmellor approved these changes

@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 is a code owner

+1 more reviewer

@Cyrilvallez Cyrilvallez Cyrilvallez left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation frontend multi-modality Related to multi-modality (#4194) new-model Requests to new models ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Milestone

v0.10.0

Development

Successfully merging this pull request may close these issues.

5 participants

@Isotr0py @hmellor @DarkLight1337 @Cyrilvallez @vllm-bot