VLMs: major clean up 🧼

[zucchini-nlp]

What does this PR do?

We have updated all the configs for VLMs on the hub so this PR removes legacy path for models, as it has been there for already 3 releases from v4.44. Also it fixes some stuff that broke on the way, like generating from only text input in LLaVA models

For Video-LLaVA the hub configs cannot be updated as the hub owner has been silent for several mmonths already. And since there is only one model with such architecture, we can hardcode the default values for patch_num and also remove the legacy path

fixes #34824, fixes #35169 and fixes #35450, fixes #35424

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

I don't think we need this, we deprecated the legacy path, we can just remove it now no?
I don't remember what we said for for 4.46 but better to go with non legacy now if we can!

[zucchini-nlp]

We can remove it after updating the files on the hub and that mean we also need to change warning to error so users have chance to see what is the reason for failure.

I think the earliest we can remove is next release, because the blocking PR will prob be merged next week. After that I will take time to update all hub configs. Maybe then we'll wait for the blocking PR and remove all deprecation warnings?

[ArthurZucker]

Sounds good, let's wait a bit!

[zucchini-nlp]

@ArthurZucker i think this can be review now :)

[lcxrocks]

Any updates on this PR? Looking forward to getting llava-hf/llava-v1.6-mistral-7b-hf running for inference without images.
Related to this issue on HF.

[ArthurZucker]

Most welcome, related to #35534 were the same is done!

[ArthurZucker]

O o 👁️ 👁️

[zucchini-nlp]

Triggered slow tests on some models, will merge when those pass (or at least it doesn't add up more failing tests...)

[Rocketknight1]

This is amazing! Let me know when it's merged so I can rebase Pixtral onto it

[zucchini-nlp]

Oke, so BLIP models apparently have one extra model class which was not modified for some reason, and their official checkpoints on the hub are also not updated. Therefore, I am not adding BLIP to this PR, it will only remove legacy from LLaVAs

Slow tests are passing on my end if compared to the main branch, there are tests failing on main due to tiny logit inconsistencies. I believe it might also be my setup/hardware, we are trying to match the runner's outputs usually

I will merge this in a one hour, just last time slow test runs and done