Request to implement FreeNoise, a new diffusion scheduler

[arthur-qiu]

Model/Pipeline/Scheduler description

FreeNoise is a tuning-free and time-efficient paradigm for longer video generation based on pretrained video diffusion models. In other words, it is another free lunch that can generate longer videos given multiple prompts, with negligible time cost overhead.

Open source status

• The model implementation is available
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

• Project: http://haonanqiu.com/projects/FreeNoise.html
• Paper: https://arxiv.org/abs/2310.15169
• Code: https://github.com/arthur-qiu/LongerCrafter

[patrickvonplaten]

Sounds like a cool idea!

[patrickvonplaten]

cc @DN6 here

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[arthur-qiu]

It can be a plug-in of AnimateDiff: https://github.com/arthur-qiu/FreeNoise-AnimateDiff

[arthur-qiu]

It can be a plug-in of AnimateDiff: https://github.com/arthur-qiu/FreeNoise-AnimateDiff

@DN6 Could you help to apply it to AnimateDiff in diffusers?

[DN6]

Hi @arthur-qiu Is FreeNoise a new type of model or scheduler? I noticed that there is a checkpoint in your repo to load it?

[arthur-qiu]

Hi @arthur-qiu Is FreeNoise a new type of model or scheduler? I noticed that there is a checkpoint in your repo to load it?

Hi @DN6, FreeNoise is a kind of scheduler. That is, you can add several lines of code to AnimateDiff, and then it can generate longer videos without any additional training/tuning. The checkpoint is used for the original model (can be AnimateDiff, VideoCrafter1, or other video models).

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[DN6]

Hi @arthur-qiu. Sorry for the delay here. I do think FreeNoise is a good candidate to add to AnimateDiff.

Are the major changes related to the temporal transformer
https://github.com/arthur-qiu/FreeNoise-AnimateDiff/blob/e01d82233c595ce22f1a5eba487911c345ce7b5b/animatediff/models/motion_module.py#L251

And creating the latents?
https://github.com/arthur-qiu/FreeNoise-AnimateDiff/blob/e01d82233c595ce22f1a5eba487911c345ce7b5b/animatediff/pipelines/pipeline_animation.py#L290

I will look into how we could integrate this.