Implementation of TRT wrapping via inference.json

[borisfom]

Sample implementation of using new trt_wrap() from MONAI
Depends on Project-MONAI/MONAI#7990

[borisfom]

I observe 2.5x speedup with FP16 TRT for a small sample (28.57it/s vs 11it/s). Did not check the accuracy.

[borisfom]

@binliunls : I have merged origin/dev and fixed brats_mri_generative_diffusion and brats_mri_axial_slices_generative_diffusion. Note: for brats_mri_axial_slices_generative_diffusion, default
python -m monai.bundle run --config_file "['configs/inference.json', 'configs/inference_trt.json']" does not get to the diffusion forward, so no TRT is generated, and there may be issues with it. Let me know how to run it so that the diffusion actually gets executed.

[binliunls]

@binliunls : I have merged origin/dev and fixed brats_mri_generative_diffusion and brats_mri_axial_slices_generative_diffusion. Note: for brats_mri_axial_slices_generative_diffusion, default python -m monai.bundle run --config_file "['configs/inference.json', 'configs/inference_trt.json']" does not get to the diffusion forward, so no TRT is generated, and there may be issues with it. Let me know how to run it so that the diffusion actually gets executed.

Hi @borisfom ,
Thank you for your update. As shown in the inference config file, you should compile the diffusion and autoencoder with TRT instead of network, since the network parameter is not defined in either bundle. Please check if you can compile both diffusion and autoencoder.

Thanks,
Bin

[borisfom]

@binliunls : Thanks, you are correct! Works as expected now.

[yiheng-wang-nv]

/build

[yiheng-wang-nv]

/build

[yiheng-wang-nv]

/build

[yiheng-wang-nv]

/build

[yiheng-wang-nv]

/build

[yiheng-wang-nv]

/build