[Community Pipeline] image2image inpainting?

[batrlatom]

Is your feature request related to a problem? Please describe.
Imagine you have two images defined, a logo and a t-shirt. we want to apply the logo to the t-shirt in such a way that it matches the physical surface of the t-shirt and the logo deforms correctly.

Describe the solution you'd like
How it should look like can be viewed in the images:
https://ibb.co/HtGGsC2 + https://ibb.co/6mFc8QL -> https://ibb.co/Y7h2t8b

Describe alternatives you've considered
I was trying to predict the surface of the t-shirt with pose predictor and place the logo onto it via affine transformation.
Results are questionable and in the end it is not scalable to other objects

Additional context

[patrickvonplaten]

Interesting! That's a cool feature request!

I think we could already solve this one with the inpainting checkpoint.
Currently it works as follows:

Now you could simply overlap your logo with the t-shirt and then mask out the corners of the logo and let stable diffusion predict the "transition".

Maybe you could already try it with the demo here:
https://huggingface.co/spaces/runwayml/stable-diffusion-inpainting

Overall this could be a really cool community pipeline addition though, e.g.:

Instead of image and masked_image you can supply a third inner_image which has to fit in the white area of the masked_image and then the image simple becomes image overlapped by inner_image and the masked_image becomes masked_image

[vvvm23]

@patrickvonplaten is there anyone working on this? If not, I would like to give it a go.

[patrickvonplaten]

Please feel free to give it a go @vvvm23 :-)

[vvvm23]

@patrickvonplaten I feel the simplest way to do this would just be to copy diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_inpaint.py, add the image overlapping, and then add checks to make sure inner_image fits inside mask_image. Would you agree?

Secondly, how would you go about checking if inner_image fits inside the white area of mask_image? If the mask was a solid region, this would be a simple check against non-transparent regions of inner_image. However, for the use case you described with "masking out the corners of the logo", this wouldn't work.

To solve that, I could either:

• Make a copy of mask_image and perform a flood fill, and use that to do the check.
• Don't check at all, trust the user to mask correctly :)

[vvvm23]

Some in-progress results:

...yes that is a long long leg.

[patrickvonplaten]

Agree @vvvm23 !

[vvvm23]

Hi @patrickvonplaten

The above pull request should implement what we want for PIL image inputs.
However, during development I noticed that the reference implementation (diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_inpaint.py) doesn't seem to support torch.Tensor inputs despite the docstrings saying otherwise. It should fail here.

For the time being, I left my implementation the same, let me know what you think or if I am wrong.

[patrickvonplaten]

There might very well be a bug in the official inpainting pipeline - feel free to open a PR to solve it :-)

[vvvm23]

#1128 will fix that. I haven't made one for this pipeline yet though.