[Stable Diffusion Inpaint] Fix incorrect noise addition

[ghost]

Fix a bug of incorrect timestep noise addition to latents.
Introduce addition which improves results when using large number of iterations.
It may also improve results for small number of iterations, doesn't make things worse.

edit: also fix the mask

edit2:
change strength default, strength < 1.0 means img2img for inpainting region, should be expressly desired by the user.
results will be very poor if the user is not aiming for img2img - prompt conflict etc...

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[ghost]

Test appears to hardcode inpainting results??

[leszekhanusz]

Maybe you could also incorporate the modifications made here by @nagolinc so that the white part of the mask is not used at the beginning.
It has been discussed in PR #261

[leszekhanusz]

But even with those modifications, still no meat 😄

[ghost]

Maybe you could also incorporate the modifications made here by @nagolinc so that the white part of the mask is not used at the beginning. It has been discussed in PR #261

he may be doing something incorrect or perhaps unnecessary with strength and init_latents there.

Your particular problem I suspect is a logic problem, you have incorrect mask processing somewhere. It would be peculiar for the whites to survive the noising process.

See:
latents = init_latents_proper * mask + latents * (1-mask)

this means that anywhere the mask is 0.0 the original image information is destroyed at that value.

having a mask with values other than 0.0 or 1.0 is probably undefined behavior, I think. Blending latents seems like it would be a bad idea. And there would be opportunity there for original image to bleed through via blending with a bad mask.

[leszekhanusz]

Your particular problem I suspect is a logic problem, you have incorrect mask processing somewhere. It would be peculiar for the whites to survive the noising process.

Please try to reproduce with the images examples in PR #261, you'll see this problem is still there and goes away with the modifications from @nagolinc (it does not mean that it's really better, we are still missing something imho).

See: latents = init_latents_proper * mask + latents * (1-mask)
this means that anywhere the mask is 0.0 the original image information is destroyed at that value.

Yes, that's what we want, but the problem probably comes from the start with the init_latents which contain the original image information and is transmitted at each iteration in the latents * (1-mask) part.
That's why @nagolinc added the line init_latents_noised=init_latents*(mask)+rand_latents*(1-mask) at the beginning so that even in the initial latents, there is absolutely no original image information in the parts that we want to inpaint.

[ghost]

If after 50 steps of noise to the initial latents some meaningful information from the original survives, it may have implications for noise scheduling, mainly that it needs to be improved for all applications and not just inpainting.

Initial t-step is usually seeded with noise, and if it has such a large unintended effect on the output that could be a problem.

alternatively, there may be a bug in scheduler.add_noise().

[patrickvonplaten]

@anton-l @patil-suraj could you take a look here? :-)

[leszekhanusz]

I made a web frontend using Vue to test the inpainting of Stable Diffusion with diffusers.
It could help to test the inpainting pipelines faster.
It is available here and looks like this for now
It is still a work in progress but you can already generate images and draw inpainting mask to generate editions. Please tell me what you think.

[patrickvonplaten]

IMO for in-painting we should more or less just follow wants been shown here: https://github.com/CompVis/stable-diffusion/blob/main/scripts/inpaint.py

@patil-suraj @anton-l - is that more or less how the official script looks like?

[patil-suraj]

Thanks a lot for the pr @jackloomen !

The changes look good. Could you also update the tests here accordingly ?

https://github.com/huggingface/diffusers/blob/main/tests/test_pipelines.py#L592

[patil-suraj]

could we maybe the just convert the mask to bool like this

mask = torch.from_numpy(mask).bool()
maks = (~mask).long()

[leszekhanusz]

In the stable diffusion repo, it looks like this:

mask[mask < 0.5] = 0
mask[mask >= 0.5] = 1

[ghost]

@patil-suraj I won't be able to compute expected_slice for some days, traveling.
I changed the mask in accordance to the stable diffusion repo.

edit: i will see if the github test details show expected_slice and use that.

[patrickvonplaten]

Hey @jackloomen,

Sorry for being so late here with our reply.
Stability AI will soon release a new inpainting checkpoint so that we will then adopt CompVis/Stable-Diffusion's way of doing in-painting to the main "Inpainting" class here. The current version is more of an experimental hack than an official implementation so we think it makes most sense to not change it too much.

We've played around with the implementation of this PR and it seems that it's not strictly better for all examples. Would it be ok for you to instead add your pipeline script to the community folder: https://github.com/huggingface/diffusers/tree/main/examples/community ? :-)

[nagolinc]

Stability AI will soon release a new inpainting checkpoint

👀