Skip to content

Add InstructPix2Pix pipeline #2040

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 31 commits into from
Jan 20, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
38cc262
being pix2pix
patil-suraj Jan 19, 2023
f6cea81
ifx
patil-suraj Jan 19, 2023
8a84549
cfg image_latents
patil-suraj Jan 19, 2023
7ae9fd5
fix some docstr
patil-suraj Jan 19, 2023
b805775
fix
patil-suraj Jan 19, 2023
53f9eb5
fix
patil-suraj Jan 19, 2023
4c6d006
hack
patil-suraj Jan 20, 2023
6b84514
fix
patil-suraj Jan 20, 2023
f91e232
Apply suggestions from code review
patil-suraj Jan 20, 2023
a5bfb04
add comments to explain the hack
patil-suraj Jan 20, 2023
f624b3f
Merge branch 'pix2pix' of https://github.com/huggingface/diffusers in…
patil-suraj Jan 20, 2023
8941296
move __call__ to the top
patil-suraj Jan 20, 2023
62ee0ee
doc
patil-suraj Jan 20, 2023
3c5953a
remove height and width
patil-suraj Jan 20, 2023
3a01b54
remove depreications
patil-suraj Jan 20, 2023
74890b0
fix doc str
patil-suraj Jan 20, 2023
c4ab485
quality
patil-suraj Jan 20, 2023
297b8ca
fast tests
patil-suraj Jan 20, 2023
61b09f0
chnage model id
patil-suraj Jan 20, 2023
c4b6a77
fast tests
patil-suraj Jan 20, 2023
db37ae2
fix test
patil-suraj Jan 20, 2023
c2d1088
address Pedro's comments
patil-suraj Jan 20, 2023
67ba0ba
copyright
patil-suraj Jan 20, 2023
0a39f9d
Simple doc page.
pcuenca Jan 20, 2023
83b81a3
Apply suggestions from code review
patil-suraj Jan 20, 2023
685ff5d
style
patil-suraj Jan 20, 2023
93cd764
Remove import
pcuenca Jan 20, 2023
16b9400
address some review comments
patil-suraj Jan 20, 2023
2382a65
Merge branch 'pix2pix' of https://github.com/huggingface/diffusers in…
patil-suraj Jan 20, 2023
a7e8448
Apply suggestions from code review
patil-suraj Jan 20, 2023
5a6e86e
style
patil-suraj Jan 20, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/source/en/_toctree.yml
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,8 @@
title: Image-Variation
- local: api/pipelines/stable_diffusion/upscale
title: Super-Resolution
- local: api/pipelines/stable_diffusion/pix2pix
title: InstructPix2Pix
title: Stable Diffusion
- local: api/pipelines/stable_diffusion_2
title: Stable Diffusion 2
Expand Down
1 change: 1 addition & 0 deletions docs/source/en/api/pipelines/stable_diffusion/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ For more details about how Stable Diffusion works and how it differs from the ba
| [StableDiffusionDepth2ImgPipeline](./depth2img) | **Experimental** – *Depth-to-Image Text-Guided Generation * | | Coming soon
| [StableDiffusionImageVariationPipeline](./image_variation) | **Experimental** – *Image Variation Generation * | | [🤗 Stable Diffusion Image Variations](https://huggingface.co/spaces/lambdalabs/stable-diffusion-image-variations)
| [StableDiffusionUpscalePipeline](./upscale) | **Experimental** – *Text-Guided Image Super-Resolution * | | Coming soon
| [StableDiffusionInstructPix2PixPipeline](./pix2pix) | **Experimental** – *Text-Based Image Editing * | | [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://huggingface.co/spaces/timbrooks/instruct-pix2pix)



Expand Down
72 changes: 72 additions & 0 deletions docs/source/en/api/pipelines/stable_diffusion/pix2pix.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
<!--Copyright 2023 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
-->

# InstructPix2Pix: Learning to Follow Image Editing Instructions

## Overview

[InstructPix2Pix: Learning to Follow Image Editing Instructions](https://arxiv.org/abs/2211.09800) by Tim Brooks, Aleksander Holynski and Alexei A. Efros.

The abstract of the paper is the following:

*We propose a method for editing images from human instructions: given an input image and a written instruction that tells the model what to do, our model follows these instructions to edit the image. To obtain training data for this problem, we combine the knowledge of two large pretrained models -- a language model (GPT-3) and a text-to-image model (Stable Diffusion) -- to generate a large dataset of image editing examples. Our conditional diffusion model, InstructPix2Pix, is trained on our generated data, and generalizes to real images and user-written instructions at inference time. Since it performs edits in the forward pass and does not require per example fine-tuning or inversion, our model edits images quickly, in a matter of seconds. We show compelling editing results for a diverse collection of input images and written instructions.*

Resources:

* [Project Page](https://www.timothybrooks.com/instruct-pix2pix).
* [Paper](https://arxiv.org/abs/2211.09800).
* [Original Code](https://github.com/timothybrooks/instruct-pix2pix).
* [Demo](https://huggingface.co/spaces/timbrooks/instruct-pix2pix).


## Available Pipelines:

| Pipeline | Tasks | Demo
|---|---|:---:|
| [StableDiffusionInstructPix2PixPipeline](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_instruct_pix2pix.py) | *Text-Based Image Editing* | [🤗 Space](https://huggingface.co/spaces/timbrooks/instruct-pix2pix) |

<!-- TODO: add Colab -->

## Usage example

```python
import PIL
import requests
import torch
from diffusers import StableDiffusionInstructPix2PixPipeline

model_id = "timbrooks/instruct-pix2pix"
pipe = StableDiffusionInstructPix2PixPipeline.from_pretrained(
model_id, torch_dtype=torch.float16, revision="fp16", safety_checker=None
).to("cuda")

url = "https://raw.githubusercontent.com/timothybrooks/instruct-pix2pix/main/imgs/example.jpg"


def download_image(url):
image = PIL.Image.open(requests.get(url, stream=True).raw)
image = PIL.ImageOps.exif_transpose(image)
image = image.convert("RGB")
return image


image = download_image(url)

prompt = "turn him into a cyborg"
images = pipe(prompt, image=image, num_inference_steps=10, guidance_scale=1.1, image_guidance_scale=1).images
images[0].save("david_cyborg.png")
```

## StableDiffusionInstructPix2PixPipeline
[[autodoc]] StableDiffusionInstructPix2PixPipeline
- __call__
- all
1 change: 1 addition & 0 deletions src/diffusers/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@
StableDiffusionImg2ImgPipeline,
StableDiffusionInpaintPipeline,
StableDiffusionInpaintPipelineLegacy,
StableDiffusionInstructPix2PixPipeline,
StableDiffusionPipeline,
StableDiffusionPipelineSafe,
StableDiffusionUpscalePipeline,
Expand Down
1 change: 1 addition & 0 deletions src/diffusers/pipelines/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@
StableDiffusionImg2ImgPipeline,
StableDiffusionInpaintPipeline,
StableDiffusionInpaintPipelineLegacy,
StableDiffusionInstructPix2PixPipeline,
StableDiffusionPipeline,
StableDiffusionUpscalePipeline,
)
Expand Down
1 change: 1 addition & 0 deletions src/diffusers/pipelines/stable_diffusion/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ class StableDiffusionPipelineOutput(BaseOutput):
from .pipeline_stable_diffusion_img2img import StableDiffusionImg2ImgPipeline
from .pipeline_stable_diffusion_inpaint import StableDiffusionInpaintPipeline
from .pipeline_stable_diffusion_inpaint_legacy import StableDiffusionInpaintPipelineLegacy
from .pipeline_stable_diffusion_instruct_pix2pix import StableDiffusionInstructPix2PixPipeline
from .pipeline_stable_diffusion_upscale import StableDiffusionUpscalePipeline
from .safety_checker import StableDiffusionSafetyChecker

Expand Down
Loading