Skip to content

Commit d0f2582

Browse files
authored
[docs] Update community pipeline docs (#2989)
* update community pipeline docs * fix formatting * explain sharing workflows
1 parent 3eaead0 commit d0f2582

File tree

4 files changed

+106
-163
lines changed

4 files changed

+106
-163
lines changed

docs/source/en/_toctree.yml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@
2525
- local: using-diffusers/schedulers
2626
title: Load and compare different schedulers
2727
- local: using-diffusers/custom_pipeline_overview
28-
title: Load and add custom pipelines
28+
title: Load community pipelines
2929
- local: using-diffusers/kerascv
3030
title: Load KerasCV Stable Diffusion checkpoints
3131
title: Loading & Hub
@@ -47,9 +47,9 @@
4747
- local: using-diffusers/reproducibility
4848
title: Create reproducible pipelines
4949
- local: using-diffusers/custom_pipeline_examples
50-
title: Community Pipelines
50+
title: Community pipelines
5151
- local: using-diffusers/contribute_pipeline
52-
title: How to contribute a Pipeline
52+
title: How to contribute a community pipeline
5353
- local: using-diffusers/using_safetensors
5454
title: Using safetensors
5555
- local: using-diffusers/stable_diffusion_jax_how_to

docs/source/en/using-diffusers/contribute_pipeline.mdx

Lines changed: 89 additions & 77 deletions
Original file line numberDiff line numberDiff line change
@@ -10,30 +10,21 @@ an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express o
1010
specific language governing permissions and limitations under the License.
1111
-->
1212

13-
# How to build a community pipeline
13+
# How to contribute a community pipeline
1414

15-
*Note*: this page was built from the GitHub Issue on Community Pipelines [#841](https://github.com/huggingface/diffusers/issues/841).
15+
<Tip>
1616

17-
Let's make an example!
18-
Say you want to define a pipeline that just does a single forward pass to a U-Net and then calls a scheduler only once (Note, this doesn't make any sense from a scientific point of view, but only represents an example of how things work under the hood).
17+
💡 Take a look at GitHub Issue [#841](https://github.com/huggingface/diffusers/issues/841) for more context about why we're adding community pipelines to help everyone easily share their work without being slowed down.
1918

20-
Cool! So you open your favorite IDE and start creating your pipeline 💻.
21-
First, what model weights and configurations do we need?
22-
We have a U-Net and a scheduler, so our pipeline should take a U-Net and a scheduler as an argument.
23-
Also, as stated above, you'd like to be able to load weights and the scheduler config for Hub and share your code with others, so we'll inherit from `DiffusionPipeline`:
19+
</Tip>
2420

25-
```python
26-
from diffusers import DiffusionPipeline
27-
import torch
21+
Community pipelines allow you to add any additional features you'd like on top of the [`DiffusionPipeline`]. The main benefit of building on top of the `DiffusionPipeline` is anyone can load and use your pipeline by only adding one more argument, making it super easy for the community to access.
2822

23+
This guide will show you how to create a community pipeline and explain how they work. To keep things simple, you'll create a "one-step" pipeline where the `UNet` does a single forward pass and calls the scheduler once.
2924

30-
class UnetSchedulerOneForwardPipeline(DiffusionPipeline):
31-
def __init__(self, unet, scheduler):
32-
super().__init__()
33-
```
25+
## Initialize the pipeline
3426

35-
Now, we must save the `unet` and `scheduler` in a config file so that you can save your pipeline with `save_pretrained`.
36-
Therefore, make sure you add every component that is save-able to the `register_modules` function:
27+
You should start by creating a `one_step_unet.py` file for your community pipeline. In this file, create a pipeline class that inherits from the [`DiffusionPipeline`] to be able to load model weights and the scheduler configuration from the Hub. The one-step pipeline needs a `UNet` and a scheduler, so you'll need to add these as arguments to the `__init__` function:
3728

3829
```python
3930
from diffusers import DiffusionPipeline
@@ -43,39 +34,54 @@ import torch
4334
class UnetSchedulerOneForwardPipeline(DiffusionPipeline):
4435
def __init__(self, unet, scheduler):
4536
super().__init__()
37+
```
38+
39+
To ensure your pipeline and its components (`unet` and `scheduler`) can be saved with [`~DiffusionPipeline.save_pretrained`], add them to the `register_modules` function:
40+
41+
```diff
42+
from diffusers import DiffusionPipeline
43+
import torch
44+
45+
class UnetSchedulerOneForwardPipeline(DiffusionPipeline):
46+
def __init__(self, unet, scheduler):
47+
super().__init__()
4648

47-
self.register_modules(unet=unet, scheduler=scheduler)
49+
+ self.register_modules(unet=unet, scheduler=scheduler)
4850
```
4951

50-
Cool, the init is done! 🔥 Now, let's go into the forward pass, which we recommend defining as `__call__` . Here you're given all the creative freedom there is. For our amazing "one-step" pipeline, we simply create a random image and call the unet once and the scheduler once:
52+
Cool, the `__init__` step is done and you can move to the forward pass now! 🔥
5153

52-
```python
53-
from diffusers import DiffusionPipeline
54-
import torch
54+
## Define the forward pass
5555

56+
In the forward pass, which we recommend defining as `__call__`, you have complete creative freedom to add whatever feature you'd like. For our amazing one-step pipeline, create a random image and only call the `unet` and `scheduler` once by setting `timestep=1`:
5657

57-
class UnetSchedulerOneForwardPipeline(DiffusionPipeline):
58-
def __init__(self, unet, scheduler):
59-
super().__init__()
58+
```diff
59+
from diffusers import DiffusionPipeline
60+
import torch
61+
62+
63+
class UnetSchedulerOneForwardPipeline(DiffusionPipeline):
64+
def __init__(self, unet, scheduler):
65+
super().__init__()
6066

61-
self.register_modules(unet=unet, scheduler=scheduler)
67+
self.register_modules(unet=unet, scheduler=scheduler)
6268

63-
def __call__(self):
64-
image = torch.randn(
65-
(1, self.unet.config.in_channels, self.unet.config.sample_size, self.unet.config.sample_size),
66-
)
67-
timestep = 1
69+
+ def __call__(self):
70+
+ image = torch.randn(
71+
+ (1, self.unet.config.in_channels, self.unet.config.sample_size, self.unet.config.sample_size),
72+
+ )
73+
+ timestep = 1
6874

69-
model_output = self.unet(image, timestep).sample
70-
scheduler_output = self.scheduler.step(model_output, timestep, image).prev_sample
75+
+ model_output = self.unet(image, timestep).sample
76+
+ scheduler_output = self.scheduler.step(model_output, timestep, image).prev_sample
7177

72-
return scheduler_output
78+
+ return scheduler_output
7379
```
7480

75-
Cool, that's it! 🚀 You can now run this pipeline by passing a `unet` and a `scheduler` to the init:
81+
That's it! 🚀 You can now run this pipeline by passing a `unet` and `scheduler` to it:
7682

7783
```python
78-
from diffusers import DDPMScheduler, Unet2DModel
84+
from diffusers import DDPMScheduler, UNet2DModel
7985

8086
scheduler = DDPMScheduler()
8187
unet = UNet2DModel()
@@ -85,71 +91,80 @@ pipeline = UnetSchedulerOneForwardPipeline(unet=unet, scheduler=scheduler)
8591
output = pipeline()
8692
```
8793

88-
But what's even better is that you can load pre-existing weights into the pipeline if they match exactly your pipeline structure. This is e.g. the case for [https://huggingface.co/google/ddpm-cifar10-32](https://huggingface.co/google/ddpm-cifar10-32) so that we can do the following:
94+
But what's even better is you can load pre-existing weights into the pipeline if the pipeline structure is identical. For example, you can load the [`google/ddpm-cifar10-32`](https://huggingface.co/google/ddpm-cifar10-32) weights into the one-step pipeline:
8995

9096
```python
9197
pipeline = UnetSchedulerOneForwardPipeline.from_pretrained("google/ddpm-cifar10-32")
9298

9399
output = pipeline()
94100
```
95101

96-
We want to share this amazing pipeline with the community, so we would open a PR request to add the following code under `one_step_unet.py` to [https://github.com/huggingface/diffusers/tree/main/examples/community](https://github.com/huggingface/diffusers/tree/main/examples/community) .
97-
98-
```python
99-
from diffusers import DiffusionPipeline
100-
import torch
101-
102+
## Share your pipeline
102103

103-
class UnetSchedulerOneForwardPipeline(DiffusionPipeline):
104-
def __init__(self, unet, scheduler):
105-
super().__init__()
104+
Open a Pull Request on the 🧨 Diffusers [repository](https://github.com/huggingface/diffusers) to add your awesome pipeline in `one_step_unet.py` to the [examples/community](https://github.com/huggingface/diffusers/tree/main/examples/community) subfolder.
106105

107-
self.register_modules(unet=unet, scheduler=scheduler)
106+
Once it is merged, anyone with `diffusers >= 0.4.0` installed can use this pipeline magically 🪄 by specifying it in the `custom_pipeline` argument:
108107

109-
def __call__(self):
110-
image = torch.randn(
111-
(1, self.unet.config.in_channels, self.unet.config.sample_size, self.unet.config.sample_size),
112-
)
113-
timestep = 1
114-
115-
model_output = self.unet(image, timestep).sample
116-
scheduler_output = self.scheduler.step(model_output, timestep, image).prev_sample
108+
```python
109+
from diffusers import DiffusionPipeline
117110

118-
return scheduler_output
111+
pipe = DiffusionPipeline.from_pretrained("google/ddpm-cifar10-32", custom_pipeline="one_step_unet")
112+
pipe()
119113
```
120114

121-
Our amazing pipeline got merged here: [#840](https://github.com/huggingface/diffusers/pull/840).
122-
Now everybody that has `diffusers >= 0.4.0` installed can use our pipeline magically 🪄 as follows:
115+
Another way to share your community pipeline is to upload the `one_step_unet.py` file directly to your preferred [model repository](https://huggingface.co/docs/hub/models-uploading) on the Hub. Instead of specifying the `one_step_unet.py` file, pass the model repository id to the `custom_pipeline` argument:
123116

124117
```python
125118
from diffusers import DiffusionPipeline
126119

127-
pipe = DiffusionPipeline.from_pretrained("google/ddpm-cifar10-32", custom_pipeline="one_step_unet")
128-
pipe()
120+
pipeline = DiffusionPipeline.from_pretrained("google/ddpm-cifar10-32", custom_pipeline="stevhliu/one_step_unet")
129121
```
130122

131-
Another way to upload your custom_pipeline, besides sending a PR, is uploading the code that contains it to the Hugging Face Hub, [as exemplified here](https://huggingface.co/docs/diffusers/using-diffusers/custom_pipeline_overview#loading-custom-pipelines-from-the-hub).
123+
Take a look at the following table to compare the two sharing workflows to help you decide the best option for you:
124+
125+
| | GitHub community pipeline | HF Hub community pipeline |
126+
|----------------|------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------|
127+
| usage | same | same |
128+
| review process | open a Pull Request on GitHub and undergo a review process from the Diffusers team before merging; may be slower | upload directly to a Hub repository without any review; this is the fastest workflow |
129+
| visibility | included in the official Diffusers repository and documentation | included on your HF Hub profile and relies on your own usage/promotion to gain visibility |
132130

133-
**Try it out now - it works!**
131+
<Tip>
134132

135-
In general, you will want to create much more sophisticated pipelines, so we recommend looking at existing pipelines here: [https://github.com/huggingface/diffusers/tree/main/examples/community](https://github.com/huggingface/diffusers/tree/main/examples/community).
133+
💡 You can use whatever package you want in your community pipeline file - as long as the user has it installed, everything will work fine. Make sure you have one and only one pipeline class that inherits from `DiffusionPipeline` because this is automatically detected.
136134

137-
IMPORTANT:
138-
You can use whatever package you want in your community pipeline file - as long as the user has it installed, everything will work fine. Make sure you have one and only one pipeline class that inherits from `DiffusionPipeline` as this will be automatically detected.
135+
</Tip>
139136

140137
## How do community pipelines work?
141-
A community pipeline is a class that has to inherit from ['DiffusionPipeline']:
142-
and that has been added to `examples/community` [files](https://github.com/huggingface/diffusers/tree/main/examples/community).
143-
The community can load the pipeline code via the custom_pipeline argument from DiffusionPipeline. See docs [here](https://huggingface.co/docs/diffusers/api/diffusion_pipeline#diffusers.DiffusionPipeline.from_pretrained.custom_pipeline):
144138

145-
This means:
146-
The model weights and configs of the pipeline should be loaded from the `pretrained_model_name_or_path` [argument](https://huggingface.co/docs/diffusers/api/diffusion_pipeline#diffusers.DiffusionPipeline.from_pretrained.pretrained_model_name_or_path):
147-
whereas the code that powers the community pipeline is defined in a file added in [`examples/community`](https://github.com/huggingface/diffusers/tree/main/examples/community).
139+
A community pipeline is a class that inherits from [`DiffusionPipeline`] which means:
140+
141+
- It can be loaded with the [`custom_pipeline`] argument.
142+
- The model weights and scheduler configuration are loaded from [`pretrained_model_name_or_path`].
143+
- The code that implements a feature in the community pipeline is defined in a `pipeline.py` file.
144+
145+
Sometimes you can't load all the pipeline components weights from an official repository. In this case, the other components should be passed directly to the pipeline:
148146

149-
Now, it might very well be that only some of your pipeline components weights can be downloaded from an official repo.
150-
The other components should then be passed directly to init as is the case for the ClIP guidance notebook [here](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/CLIP_Guided_Stable_diffusion_with_diffusers.ipynb#scrollTo=z9Kglma6hjki).
147+
```python
148+
from diffusers import DiffusionPipeline
149+
from transformers import CLIPFeatureExtractor, CLIPModel
150+
151+
model_id = "CompVis/stable-diffusion-v1-4"
152+
clip_model_id = "laion/CLIP-ViT-B-32-laion2B-s34B-b79K"
153+
154+
feature_extractor = CLIPFeatureExtractor.from_pretrained(clip_model_id)
155+
clip_model = CLIPModel.from_pretrained(clip_model_id, torch_dtype=torch.float16)
156+
157+
pipeline = DiffusionPipeline.from_pretrained(
158+
model_id,
159+
custom_pipeline="clip_guided_stable_diffusion",
160+
clip_model=clip_model,
161+
feature_extractor=feature_extractor,
162+
scheduler=scheduler,
163+
torch_dtype=torch.float16,
164+
)
165+
```
151166

152-
The magic behind all of this is that we load the code directly from GitHub. You can check it out in more detail if you follow the functionality defined here:
167+
The magic behind community pipelines is contained in the following code. It allows the community pipeline to be loaded from GitHub or the Hub, and it'll be available to all 🧨 Diffusers packages.
153168

154169
```python
155170
# 2. Load the pipeline class, if using custom module then load it from the hub
@@ -164,6 +179,3 @@ else:
164179
diffusers_module = importlib.import_module(cls.__module__.split(".")[0])
165180
pipeline_class = getattr(diffusers_module, config_dict["_class_name"])
166181
```
167-
168-
This is why a community pipeline merged to GitHub will be directly available to all `diffusers` packages.
169-

docs/source/en/using-diffusers/custom_pipeline_examples.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express o
1010
specific language governing permissions and limitations under the License.
1111
-->
1212

13-
# Custom Pipelines
13+
# Community pipelines
1414

1515
> **For more information about community pipelines, please have a look at [this issue](https://github.com/huggingface/diffusers/issues/841).**
1616

0 commit comments

Comments
 (0)