[Prompt Diffusion] In-Context Learning Unlocked for Diffusion Models

[sayakpaul]

In short, prompt diffusion allows for in-context learning for diffusion models.

Code: https://github.com/Zhendong-Wang/Prompt-Diffusion
Models: https://huggingface.co/zhendongw/prompt-diffusion

One should be able to use https://github.com/huggingface/diffusers/blob/main/scripts/convert_original_controlnet_to_diffusers.py to get the model converted into diffusers format.

input_cond_block seems to be the new addition: https://github.com/Zhendong-Wang/Prompt-Diffusion/blob/d5737b81497c17af8d1e0b242f03dadab93bec1c/cldm/cldm.py#L165. This needs to be incorporated in the conversion and in the ControlNet block.

And then it's incorporated like so:
https://github.com/Zhendong-Wang/Prompt-Diffusion/blob/d5737b81497c17af8d1e0b242f03dadab93bec1c/cldm/cldm.py#L302 (execution starts from https://github.com/Zhendong-Wang/Prompt-Diffusion/blob/d5737b81497c17af8d1e0b242f03dadab93bec1c/run_prompt_diffusion.ipynb)

Happy to help anyone willing to take this one up!

[charchit7]

Hey @sayakpaul, I'm happy to take this. I might take some time in implementing things, but I'm committed to sticking with it until it's completed.

[sayakpaul]

Maybe let's start with a community pipeline. If there's interest, we can graduate it to a core pipeline.

[charchit7]

Sure, got it.

[charchit7]

@sayakpaul Hey, just wanted to update here. Was stuck with a submission deadline. Will start with this in 1-2 day.

[iczaw]

Hello @sayakpaul, finished a quick and dirty sketch. Any suggestions or feedback would be highly appreciated: https://github.com/iczaw/diffusers-pd/blob/main/examples/community/pipeline_prompt_diffusion.py

[sayakpaul]

I think that should work. Feel free to open a PR for this. Cc: @charchit7

[charchit7]

@sayakpaul sorry for holding it this late. I am not able to get time to work on this.
Hi @iczaw nice work. Please go ahead with a PR. I am closing mine.

[iczaw]

@sayakpaul Ok.
@charchit7 Thank you!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.