Blended Diffusion

[patrickvonplaten]

Model/Pipeline/Scheduler description

https://github.com/omriav/blended-diffusion

Cool model for in-painting

Open source status

• The model implementation is available
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

Think @ydshieh is already working on it 🚀

[ydshieh]

Yes, I am on it, although somehow slow.

is this method not obvious and already implemented in SD inpainting available here?

[leszekhanusz]

The inpainting model currently implemented here is not really working correctly.
Here is a example result using their test input (steps:50, strength:0.8, guidance_scale: 7.5, prompt:"meat"):

In addition of not providing a good result in the masked part, the rest of the image which should be untouched is actually modified with noise.
On this line, the last time we apply the mask, we are using a noised version.

To obtain good results with current inpainting the prompt should be detailed and include what is outside the inpainting region, ideally it's a variation of the original prompt that generated the original image.

Also, using a noised version is the correct way to do it. But there is a bug, it should be t-1 instead of t.

[leszekhanusz]

Tried with "Cheetah on a dirt road, meat is on the floor in front of him", using also t-1 instead of t but I still don't get good results:

Try this PR:
#328

There is also something wrong with this cat, for testing I would use some generated photo of a dog with a nice prompt. then once that works move to other things.

Also be very careful with strength, it's img2img for the inpainted region. Should be 1.0 (or 0.0, i don't remember the math behind it right now) unless you actually desire img2img.

I believe you may have been doing img2img on road trying to get meat, obviously that would end poorly.

[patrickvonplaten]

Thanks a mille for the comments here @jackloomen and @leszekhanusz - that's super useful! @patil-suraj @anton-l could you here if you find time?

[ydshieh]

@jackloomen @leszekhanusz My quick understating from the above discussions - you are testing pipeline_stable_diffusion_inpaint using a test input image from blended-diffusion, right?

[patrickvonplaten]

I think this is totally unrelated to Blended diffusion @ydshieh (actually we should have moved the discussion here) Sorry for the confusion