[Feature request]: Dreambooth support for text2video diffusion models

[kabachuha]

Model/Pipeline/Scheduler description

Now that the first text2video pipeline is merged, the next frontier is to enable efficient fine-tuning of these models.

There's already Dreambooth method for the Diffusion-based text2img models, consisting of making a regularization dataset and then finetuning the model in a specific way on both the input and on the generated dataset.

Given that the nature of these models is essentially the same (as I know for maintaining the Auto1111 extension for this model), it can be enabled by shifting the Unet model from 2d to 3d and changing the class dataset pipeline from images to video. It would be really awesome to do 🙂

Open source status

• The model implementation is available
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

No response

[patrickvonplaten]

This should actually be possible indeed! cc @sayakpaul here

[ExponentialML]

I've created an implementation that should work if you want to pull from it. It's based off of the Tune-A-Video pipeline with some tweaks here and there.

https://github.com/ExponentialML/Text-To-Video-Finetuning

[sayakpaul]

This should indeed be possible.

@kabachuha @ExponentialML would anyone of you like to open a PR? We're more than happy to guide throughout :)

[silvererudite]

I'd be very happy to take a jab at this if it's okay with everyone here, @sayakpaul can you please tell me what are the components I should put in the PR/the kind of PR that I should open for this task?

[sayakpaul]

Feel free to take a stab at it.

The PR should be similar in terms of components to this one: #2478

[patrickvonplaten]

https://github.com/ExponentialML/Text-To-Video-Finetuning looks super cool!

BTW @ExponentialML would it be ok if we add a dreambooth for text-to-video fine-tuning script to diffusers' examples with you being the original author or would you prefer that we just link to your GitHub?

[sayakpaul]

@ExponentialML a friendly ping ^.

[adhikjoshi]

@ExponentialML a friendly ping ^.

IMO too many text2video models coming. We should choose best one and provide updates.

[sayakpaul]

Having a training example would still be quite useful for the community, especially for use cases where personalization is required.

IMO too many text2video models coming.

That's even true for diffusers:

• [Pipeline] Port Tune-A-Video pipeline to diffusers #2455
• [Pipeline] Add TextToVideoZeroPipeline #2954

[ExponentialML]

https://github.com/ExponentialML/Text-To-Video-Finetuning looks super cool!

BTW @ExponentialML would it be ok if we add a dreambooth for text-to-video fine-tuning script to diffusers' examples with you being the original author or would you prefer that we just link to your GitHub?

Sorry for the late response as my notifications are kind of messy :-). Sure, do as you wish.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.