[Community] FP16 ONNX produces incorrect output

[kleiti]

Describe the bug

#932 enabled conversion of the main branch FP32 model (git clone https://huggingface.co/CompVis/stable-diffusion-v1-4 -b main) to ONNX FP16. While it runs fine with OnnxStableDiffusionPipeline using DMLExecutionProvider (onnxruntime-directml==1.13.1), the produced image is just a black square.

Conversion was done with:
convert_stable_diffusion_checkpoint_to_onnx.py --model_path stable-diffusion-v1-4 --output_path fp16_sd14 --fp16

Reproduction

from diffusers import OnnxStableDiffusionPipeline

pipe = OnnxStableDiffusionPipeline.from_pretrained("fp16_sd14", provider="DmlExecutionProvider")
prompt = "viking storming a castle"
image = pipe(prompt).images[0]
image.save("viking.png")

Logs

No response

System Info

• diffusers version: 0.7.0.dev0
• Platform: Windows-10-10.0.22000-SP0
• Python version: 3.9.13
• PyTorch version (GPU?): 1.12.1+cpu (False)
• Huggingface_hub version: 0.10.0
• Transformers version: 4.22.2
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: No

[patrickvonplaten]

@anton-l could you take a look here?

[anton-l]

Hi @kleiti! Does it still produce a black image when trying other promts?
If it does, could you try initializing it with pipe = OnnxStableDiffusionPipeline.from_pretrained("fp16_sd14", provider="DmlExecutionProvider", safety_checker=None)? Might be a bug in the safety checker conversion if the black image persists for other prompts.
I don't have a windows machine set up for testing yet, so I'm not able to check myself. We'll work on windows testing next week.

[averad]

@anton-l

Windows 11
Python 3.10.8
Diffusers 0.6.0

Input
pipe = OnnxStableDiffusionPipeline.from_pretrained(model, provider="DmlExecutionProvider", safety_checker=None)

Output (Error)
pipe = OnnxStableDiffusionPipeline.from_pretrained( File "D:\ai\sd_env\lib\site-packages\diffusers\pipeline_utils.py", line 474, in from_pretrained raise ValueError( ValueError: None is of type: <class 'NoneType'>, but should be <class 'diffusers.onnx_utils.OnnxRuntimeModel'>

Workaround (Not a fix):
After initializing the pipe, call
pipe.safety_checker = lambda images, **kwargs: (images, False)

[kleiti]

Hi @kleiti! Does it still produce a black image when trying other promts?

It does, and if I add safety_checker=None, I see the same issue @averad mentions above.

To go around that, I commented out safety_checker and feature_extractor in model_index.json and pipeline_onnx_stable_diffusion.py, but that didn't help. Still outputs black image.

Glad to hear about your intention of doing Windows testing.

[averad]

Opened Issue#1138 for the OnnxStableDiffusionPipeline safety_checker=None issue.

[anton-l]

#1173 didn't help, keeping the issue open. Can't reproduce at the moment without a DirectML setup (works well on cuda and cpu though).

If anyone reading this has a hunch about what could be wrong, feel free to open a bugfix PR!

[decahedron1]

For me, it generates an image properly on CPU, but a black image on CUDA (with safety checker disabled). Seems the float16 unet is outputting all NaN on GPU.

I found 2 similar issues from onnxruntime:
microsoft/onnxruntime#6328
microsoft/onnxruntime#11384 - fixed with auto_convert_mixed_precision

Maybe the unet needs to be converted with auto_convert_mixed_precision_model_path? 🤔

[decahedron1]

I've been converting models via Colab as my system doesn't have enough VRAM, and interestingly, testing a float16 model on Colab yields correct results, but using the model on my system yields NaN. Could this issue be related to this comment then?