Is it possible to convert the onnx model to fp16 model?

[yuananf]

The torch example gives parameter revision="fp16", can onnx model do the same optimization? Current onnx inference(using CUDAExecutionProvider) is slower than torch version, and used more gpu memory than torch version(12G vs 4G).

[anton-l]

Hi @yuananf! At the moment the onnx pipeline is less optimized than its pytorch counterpart, so all computation happens in float32 and there's overhead due to cpu-gpu tensor copies in the inference sampling loop.
For now only the CPU runtime offers a significant speedup over pytorch, but we're working with the onnxruntime team on a GPU revamp. If you see something obvious that could improve performace on GPU: feel free to open a PR and we'll integrate it! :)

[yuananf]

Thank you for your response!

[yuananf]

Hi @yuananf! At the moment the onnx pipeline is less optimized than its pytorch counterpart, so all computation happens in float32 and there's overhead due to cpu-gpu tensor copies in the inference sampling loop. For now only the CPU runtime offers a significant speedup over pytorch, but we're working with the onnxruntime team on a GPU revamp. If you see something obvious that could improve performace on GPU: feel free to open a PR and we'll integrate it! :)

To avoid data copy between cpu and gpu, onnxruntime provided IOBinding feature. https://onnxruntime.ai/docs/api/python/api_summary.html#data-on-device

[wareya]

It's possible to convert individual .onnx things to fp16 with this script from onnxconverter: https://github.com/microsoft/onnxconverter-common/blob/master/onnxconverter_common/auto_mixed_precision_model_path.py. I need to do this to get 512x512 stable diffusion to fit in dedicated VRAM on my GPU (an 8gb rx 480), giving me 2 seconds per iteration instead of 16. However the process for doing this is very cumbersome, because you need to feed each stage the exact inputs that it typically expects (I ended up copy/pasting the onnx pipeline file and added in code to copy the inputs to each stage), and it's very slow, because it essentially brute forces which parts it can convert to fp16 without causing the model's output to significantly change. Also, at least on my system, I had to edit the auto mixed precision script to deal with the case of stale file handles by force deleting said files instead of trying to delete them normally (it would crash with a file access violation because they were still open in stale handles).

[tdeboissiere]

@wareya could you share the details of your scripts / conversion process ?

[wareya]

First, get the full-precision onnx model locally from the onnx exporter (convert_stable_diffusion_checkpoint_to_onnx.py). For example:

python convert_stable_diffusion_checkpoint_to_onnx.py --model_path="CompVis/stable-diffusion-v1-4" --output_path="./stable_diffusion_onnx" --height=512 --width=512

Then modify pipeline_stable_diffusion_onnx.py to call out to auto_convert_mixed_precision_model_path with the relevant paths and inputs at the right points in the pipeline. I modified it like so: https://gist.github.com/wareya/0d5d111b1e2448a3e99e8be2b39fbcf1 (I've modified this since the last time I ran it, so it might be slightly broken. YMMV. Also, this is extremely hacky and bad and you shouldn't encourage other people to do it this way or do it this way in any release pipelines, but it was the fastest way for me to get it done locally.)

However, on my system, this crashes inside of auto_mixed_precision_model_path.py because it tries to delete files that it still has open. Might be a bug in the exact version of the onnx runtime that I'm running (I'm running a nightly version). To work around it, I modify _clean_output_folder in auto_mixed_precision_model_path.py like this (this is evil and might only work on Windows 10, not other versions of windows, and will not work on non-windows OSs):

    if os.path.exists(tmp_tensor_path):
        try:
            os.remove(tmp_tensor_path)
        except:
            try:
                tmp_tensor_path = tmp_tensor_path.replace("/", "\\")
                print(f"force deleting {tmp_tensor_path}")
                os.system(f'cmd /c "del /f {tmp_tensor_path}"')
            except:
                print("no idea what broke here but something did!")

[tianleiwu]

Latest script can be found here: https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/models/stable_diffusion/README.md

Example script to convert FP32 to FP16:

 # You can clone the source code of onnxruntime to run this script as the following:
 #    git clone https://github.com/microsoft/onnxruntime
 #    cd onnxruntime/onnxruntime/python/tools/transformers 
 #    save this script to the directory as sd_fp16.py. Modify the root_dir if needed.
 #    python sd_fp16.py
    
import os
import shutil
import onnx
from onnxruntime.transformers.optimizer import optimize_model

# root directory of the onnx pipeline data files
root_dir = "./sd_onnx"

for name in ["unet", "vae_encoder", "vae_decoder", "text_encoder", "safety_checker"]:
    onnx_model_path = f"{root_dir}/{name}/model.onnx"

    # The following will fuse LayerNormalization and Gelu. Do it before fp16 conversion, otherwise they cannot be fused later.
    # Right now, onnxruntime does not save >2GB model so we use script to optimize unet instead.
    m = optimize_model(
        onnx_model_path,
        model_type="bert",
        num_heads=0,
        hidden_size=0,
        opt_level=0,
        optimization_options=None,
        use_gpu=False,
    )

    # Use op_bloack_list to force some operators to compute in FP32.
    # TODO: might need some tuning to add more operators to op_bloack_list to reduce accuracy loss.
    if name == "safety_checker":
        m.convert_float_to_float16(op_block_list=["Where"])
    else:
        m.convert_float_to_float16(op_block_list=["RandomNormalLike"])

    # Overwrite existing models. You can change it to another directory but need copy other files like tokenizer manually.
    optimized_model_path = f"{root_dir}/{name}/model.onnx"
    output_dir = os.path.dirname(optimized_model_path)
    shutil.rmtree(output_dir)
    os.mkdir(output_dir)

    onnx.save_model(m.model, optimized_model_path)

To get best performance, please set providers like the following:

   providers = [("CUDAExecutionProvider", {"cudnn_conv_use_max_workspace": '1'})]

See https://onnxruntime.ai/docs/performance/tune-performance.html#convolution-heavy-models-and-the-cuda-ep for more info.

Latency (Seconds per Query) for GPU

Stable Diffusion Pipeline (text to 512x512 image)	T4	V100	A100
PyTorch FP16	12.8	5.1	3.1
Onnx FP32	26.2	8.3	4.9
Onnx FP16	9.6	3.8	2.4

[patrickvonplaten]

Nice! @anton-l do you want to take a look here?

[cprivitere]

I tried @tianleiwu 's script and all my results, while twice as fast, are very washed out. I'm also getting a lot of all black box results. This was on an AMD card using the DML provider on windows. The non converted-to-fp16 files work fine.

[tianleiwu]

@cprivitere, thanks for the feedback. To improve accuracy, onnx model need convert to mixed precision by adding some operators (like LayerNormalization, Gelu etc) to op_block_list in the script. The list could be tuned by some A/B testing (Comparing to fp32 results) using a test set.

[cprivitere]

I've had some time since my last post to actually install and set all this up on Linux, none of the speed increases here even come close to the speed of running the fp16 models on ROCm, sadly. On a 6750XT we're talking 4.5it/s using LMS on Linux fp16 ROCm models versus 1.4it/s using LMS with the broken fp16 onnx models.

[patrickvonplaten]

Any updates here @anton-l ?

[anton-l]

FP16 models are now supported when tracing on GPU, thanks to @SkyTNT: #932
AFAIK @tianleiwu is working on a working CPU-only conversion script :)