[New Model]: HuggingFaceTB/SmolVLM2-2.2B-Instruct

[ismael-dm]

The model to consider.

HuggingFaceTB/SmolVLM2-2.2B-Instruct

The closest model vllm already supports.

HuggingFaceTB/SmolVLM-256M-Instruct

What's your difficulty of supporting the model you want?

Current error from vLLM: SmolVLMForConditionalGeneration has no vLLM implementation and the Transformers implementation is not compatible with vLLM. Try setting VLLM_USE_V1=0.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[gpcbitnik]

save the model localy with save_pretrained, open config.json, replace "architectures": [
"SmolVLMForConditionalGeneration"
], by "architectures": [
"Idefics3ForConditionalGeneration"
]
and "model_type": "smolvlm",
by "model_type": "idefics3",

now i can load the model and infer it
(i try this because the huggingface page said that the architecture is based on idefics)

[miguelalba96]

@gpcbitnik do you have the same consistent outputs compared to using model.generate?

[gpcbitnik]

Yes, for my fine tuned model on image to json task it works well and provide the same result as the inference with transformers
But i found it a bit slow, not sur if it's the model or my vllm parameters

[chaunceyjiang]

I can try to fix this issue.

[danielsyahputra]

save the model localy with save_pretrained, open config.json, replace "architectures": [ "SmolVLMForConditionalGeneration" ], by "architectures": [ "Idefics3ForConditionalGeneration" ] and "model_type": "smolvlm", by "model_type": "idefics3",

now i can load the model and infer it (i try this because the huggingface page said that the architecture is based on idefics)

Hi, I already did what you mention in here. But still got this error

BadRequestError: Error code: 400 - {'object': 'error', 'message': 'Unknown video model type: idefics3', 'type': 'BadRequestError', 'param': None, 'code': 400}

I did two approach, the first one was directly changing the value in config.json file, then the second to add a --hf-overrides to override value while serving.

vllm serve HuggingFaceTB/SmolVLM2-2.2B-Instruct --gpu-memory-utilization 0.9 --hf-overrides '{"architectures": ["Idefics3ForConditionalGeneration"], "model_type": "idefics3"}'

Did you know whats wrong with this problem?