Skip to content

Conversation

@haikmanukyan
Copy link
Contributor

@haikmanukyan haikmanukyan commented Mar 29, 2024

What does this PR do?

This PR adds HD-Painter.
HD-Painter is a training-free approach for improving text-alignment of Stable Diffusion based text-guided inpainting models.
It adds the StableDiffusionHDPainterPipeline which is an inpainting pipeline inheriting from StableDiffusionInpaintPipeline,
as well as the PAIntAAttnProcessor, an attention processor implementing the PAIntA layer from the paper.

Usage example

import torch
from diffusers import DiffusionPipeline, DDIMScheduler
from diffusers.utils import load_image, make_image_grid

pipe = DiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-2-inpainting",
    custom_pipeline="hd_painter"
)
pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config)

prompt = "wooden boat"
init_image = load_image("https://raw.githubusercontent.com/Picsart-AI-Research/HD-Painter/main/__assets__/samples/images/2.jpg")
mask_image = load_image("https://raw.githubusercontent.com/Picsart-AI-Research/HD-Painter/main/__assets__/samples/masks/2.png")

image = pipe (prompt, init_image, mask_image, use_rasg = True, use_painta = True, generator=torch.manual_seed(12345)).images[0]

make_image_grid([init_image, mask_image, image], rows=1, cols=3)

Before submitting

@sayakpaul

@sayakpaul
Copy link
Member

Thanks for contributing!

We should be able to pass custom_pipeline="..." and not the way you're showing in the OP. You can refer to the community examples REAMDE to know more about it.

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@haikmanukyan
Copy link
Contributor Author

@sayakpaul

@haikmanukyan
Copy link
Contributor Author

@sayakpaul done

@sayakpaul
Copy link
Member

Will merge once the CI is green.

@haikmanukyan
Copy link
Contributor Author

@sayakpaul I was using wrong ruff version, it should pass the check now

@sayakpaul sayakpaul merged commit 5266ab7 into huggingface:main Apr 1, 2024
@sayakpaul
Copy link
Member

Thanks for your contributions.

noskill pushed a commit to noskill/diffusers that referenced this pull request Apr 5, 2024
* add HD-Painter pipeline

* style fixing

* refactor, change doc, fix ruff

* fix docs

* used correct ruff version

---------

Co-authored-by: Hayk Manukyan <[email protected]>
Co-authored-by: Sayak Paul <[email protected]>
sayakpaul added a commit that referenced this pull request Dec 23, 2024
* add HD-Painter pipeline

* style fixing

* refactor, change doc, fix ruff

* fix docs

* used correct ruff version

---------

Co-authored-by: Hayk Manukyan <[email protected]>
Co-authored-by: Sayak Paul <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants