From ccafcf63e5e0da4b6c05d4321d94f049d439b366 Mon Sep 17 00:00:00 2001 From: Sayak Paul Date: Thu, 16 Mar 2023 11:24:55 +0530 Subject: [PATCH 1/2] minor edits to onnx and openvino docs. --- docs/source/en/optimization/onnx.mdx | 67 ++++++++++------------- docs/source/en/optimization/open_vino.mdx | 2 +- 2 files changed, 30 insertions(+), 39 deletions(-) diff --git a/docs/source/en/optimization/onnx.mdx b/docs/source/en/optimization/onnx.mdx index 6b891ef83347..28d0045500a0 100644 --- a/docs/source/en/optimization/onnx.mdx +++ b/docs/source/en/optimization/onnx.mdx @@ -13,61 +13,52 @@ specific language governing permissions and limitations under the License. # How to use the ONNX Runtime for inference -🤗 Diffusers provides a Stable Diffusion pipeline compatible with the ONNX Runtime. This allows you to run Stable Diffusion on any hardware that supports ONNX (including CPUs), and where an accelerated version of PyTorch is not available. +🤗 [Optimum](https://github.com/huggingface/optimum-intel) provides a Stable Diffusion pipeline compatible with ONNX. ## Installation -- TODO +Install 🤗 Optimum with the following command for ONNX support: + +``` +pip install optimum["onnxruntime"] +``` ## Stable Diffusion Inference -The snippet below demonstrates how to use the ONNX runtime. You need to use `OnnxStableDiffusionPipeline` instead of `StableDiffusionPipeline`. You also need to download the weights from the `onnx` branch of the repository, and indicate the runtime provider you want to use. +To load an ONNX model and run inference with the ONNX Runtime, you need to replace [`StableDiffusionPipeline`] with `ORTStableDiffusionPipeline`. In case you want to load +a PyTorch model and convert it to the ONNX format on-the-fly, you can set `export=True`. ```python -# make sure you're logged in with `huggingface-cli login` -from diffusers import OnnxStableDiffusionPipeline - -pipe = OnnxStableDiffusionPipeline.from_pretrained( - "runwayml/stable-diffusion-v1-5", - revision="onnx", - provider="CUDAExecutionProvider", -) +from optimum.onnxruntime import ORTStableDiffusionPipeline +model_id = "runwayml/stable-diffusion-v1-5" +pipe = ORTStableDiffusionPipeline.from_pretrained(model_id, export=True) prompt = "a photo of an astronaut riding a horse on mars" -image = pipe(prompt).images[0] +images = pipe(prompt).images[0] ``` -The snippet below demonstrates how to use the ONNX runtime with the Stable Diffusion upscaling pipeline. +If you want to export the pipeline in the ONNX format offline and later use it for inference, +you can use the [`optimum-cli export`](https://huggingface.co/docs/optimum/main/en/exporters/onnx/usage_guides/export_a_model#exporting-a-model-to-onnx-using-the-cli) command: -```python -from diffusers import OnnxStableDiffusionPipeline, OnnxStableDiffusionUpscalePipeline +```bash +optimum-cli export onnx --model runwayml/stable-diffusion-v1-5 sd_v15_onnx/ +``` + +Then perform inference: + +```python +from optimum.onnxruntime import ORTStableDiffusionPipeline +model_id = "sd_v15_onnx" +pipe = ORTStableDiffusionPipeline.from_pretrained(model_id) prompt = "a photo of an astronaut riding a horse on mars" -steps = 50 - -txt2img = OnnxStableDiffusionPipeline.from_pretrained( - "runwayml/stable-diffusion-v1-5", - revision="onnx", - provider="CUDAExecutionProvider", -) -small_image = txt2img( - prompt, - num_inference_steps=steps, -).images[0] - -generator = torch.manual_seed(0) -upscale = OnnxStableDiffusionUpscalePipeline.from_pretrained( - "ssube/stable-diffusion-x4-upscaler-onnx", - provider="CUDAExecutionProvider", -) -large_image = upscale( - prompt, - small_image, - generator=generator, - num_inference_steps=steps, -).images[0] +images = pipe(prompt).images[0] ``` +Notice that we didn't have to specify `export=True` above. + +You can find more examples in [optimum documentation](https://huggingface.co/docs/optimum/). + ## Known Issues - Generating multiple prompts in a batch seems to take too much memory. While we look into it, you may need to iterate instead of batching. diff --git a/docs/source/en/optimization/open_vino.mdx b/docs/source/en/optimization/open_vino.mdx index 63027835afc8..5366e86b4a54 100644 --- a/docs/source/en/optimization/open_vino.mdx +++ b/docs/source/en/optimization/open_vino.mdx @@ -36,4 +36,4 @@ prompt = "a photo of an astronaut riding a horse on mars" images = pipe(prompt).images[0] ``` -You can find more examples in [optimum documentation](https://huggingface.co/docs/optimum/intel/inference#export-and-inference-of-stable-diffusion-models). +You can find more examples (such as static reshaping and model compilation) in [optimum documentation](https://huggingface.co/docs/optimum/intel/inference#export-and-inference-of-stable-diffusion-models). From 76b5ef0cbb3b7bfb840ba8bc69ec30254b8c587c Mon Sep 17 00:00:00 2001 From: Sayak Paul Date: Thu, 16 Mar 2023 22:58:49 +0530 Subject: [PATCH 2/2] Apply suggestions from code review Co-authored-by: Ella Charlaix <80481427+echarlaix@users.noreply.github.com> --- docs/source/en/optimization/onnx.mdx | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/source/en/optimization/onnx.mdx b/docs/source/en/optimization/onnx.mdx index 28d0045500a0..6f96ba0cc194 100644 --- a/docs/source/en/optimization/onnx.mdx +++ b/docs/source/en/optimization/onnx.mdx @@ -13,11 +13,11 @@ specific language governing permissions and limitations under the License. # How to use the ONNX Runtime for inference -🤗 [Optimum](https://github.com/huggingface/optimum-intel) provides a Stable Diffusion pipeline compatible with ONNX. +🤗 [Optimum](https://github.com/huggingface/optimum) provides a Stable Diffusion pipeline compatible with ONNX Runtime. ## Installation -Install 🤗 Optimum with the following command for ONNX support: +Install 🤗 Optimum with the following command for ONNX Runtime support: ``` pip install optimum["onnxruntime"] @@ -35,6 +35,7 @@ model_id = "runwayml/stable-diffusion-v1-5" pipe = ORTStableDiffusionPipeline.from_pretrained(model_id, export=True) prompt = "a photo of an astronaut riding a horse on mars" images = pipe(prompt).images[0] +pipe.save_pretrained("./onnx-stable-diffusion-v1-5") ``` If you want to export the pipeline in the ONNX format offline and later use it for inference,