Apple MPS image generation times take much longer with 0.4.0

[FahimF]

Describe the bug

It used to take about 90 seconds to generate a 768 x 512 image with diffusers 0.3.0. I checked out the main branch with 0.4.0dev0 today and the time has increased to about 600 seconds and that too for only 20 steps, as opposed to 75 steps initially. It feels as if this build uses both the CPU and GPU and relies more on the CPU but is mostly based on gut feel, the fact that the system UI is less responsive and the fact that the Activity Monitor only shows about 50% GPU usage at most times ...

So has something changed? Performance under Apple Silicon has gotten worse though it is much more stable. Do note that 512 x 512 image generation still seems to work at about the same speed but if you try 768 x 512 or 768 x 768, the image generation slows down a lot.

Update: I was wrong about 512 x 512 not being affected. That too has slowed down but not as much. A 512 x 512 image used to take around 55 second with 0.3.0 but now it takes around 80 seconds.

Reproduction

Just run an image generation with Stable Diffusion but for an image size other than 512 x 512. I've tested for both 768 x 768 and 768 x 512 and the results were really slow.

Logs

No response

System Info

diffusers 0.4.0dev0
Apple M1 Max
Monterey 12.6

[patrickvonplaten]

Gently pinging @pcuenca here :-)

(sorry for the long wait @FahimF)

@pcuenca maybe we should post a longer "what's the current state" message in #760 ?

[pcuenca]

Hi @FahimF my apologies for being so slow! I need to seriously test the latest pre-release versions of PyTorch and try to find what the problem might be. I hope to provide a better update on Monday. Thanks for your patience, and also for your help in reporting these problems, they are very helpful even if we are slow sometimes.

[FahimF]

@pcuenca Not blaming you guys, at all! Much appreciate what you are doing since I did try working with the Tensorflow version of Stable Diffusion (because it is much faster than PyTorch and Hugging Face Diffusers at the moment) and I must confess that I much prefer working with PyTorch and Hugging Face Diffusers if I can 😄

So, really appreciate what you guys are doing and am simply reporting issues as I see them since I do work with PyTorch MPS stuff almost every day. Just not at the coding level I'd like, to be able to help out. But am trying to learn more so that I can help out .. especially since PyTorch has some open tickets for adding missing MPS functionality which are labeled as "Good first issues". But not there yet since I get distracted by working with Stable Diffusion 😄

Anyway, if you want numbers, tests, or further information, please let me know. I'd be happy to provide more info. I also just noticed that you guys are up to 0.5.1 now. I don't know if I tested the latest releases - I do know that I tested 0.4.0 after it was released and it was still slow. Will test and update this ticket if the situation has changed.

[FahimF]

@pcuenca Tried testing with 0.5.1 to see if there's a change with the speed issues but can't even get simple code to run using 0.5.1 with diffusers due to a persistent crash. Filed a separate bug report about that here: #852

[FahimF]

@pcuenca Tested with the latest diffusers (0.5.1) and the pre-slowdown version of PyTorch nightly (1.13.0.dev20220924) and it appears that diffusers is now almost as fast as it used to be. So there's some improvement over the initial 0.4.0 pre-release builds I last tested on but slightly slower than it used to be ...

But for anything other than 512 x 512 images, it's still extremely slow and that is not the behaviour you could see with diffusers 0.3.0 ... I tested generating a 768 x 512 image and that took almost 20 minutes (19 mins 17 sec) and is far worse than even my last run ☹️

To compare, I reverted back to diffusers 0.3.0 and ran an image generation for a 768 x 512 image and it completed in 58 seconds!

I monitored CPU/GPU usage for both of the above image generations and for the diffusers 0.5.1 run, the GPU usage never went above 60% while the CPU usage climbed to over 90% - 100% at times. But for diffusers 0.3.0, the CPU usage remained relatively low while the GPU usage was always around 98% during the image generation. So it appears that since 0.3.0, some tasks have switched to CPU instead of GPU and this is why things are slower?

[pcuenca]

Hi @FahimF that appears to be the case, absolutely!

I have of couple of not-yet-updates:

• Some of the workarounds we added for unsupported ops are no longer necessary with the combination of PyTorch 1.13 pre-RC and the latest macOS Ventura beta. I think most were solved by the PyTorch team, while Apple fixed part of the array size limitations that caused weird crashes and inconsistencies (some of them are still failing though). I did not notice much of a performance improvement in my tests, but I didn't test non-standard sizes. I'll do it tomorrow, and I'll post a branch in case you'd like to test too.

• The PyTorch team is hard at work trying to fix some performance degradation issues that affect MPS, see for example: [einsum] Call view instead of sum to remediate MPS regression pytorch/pytorch#87135. I'm not sure these fixes will make it into the final release, but it's looking great :)

• Another factor I need to look into is the impact of system memory size on these problems. My test computer has 64 GB of RAM, which is a lot. I've always suspected that a system with less RAM might reach a cliff of heavy degradation if it needs to page/swap. The kind of performance you describe (fast for 512, extremely slow for larger sizes) could be indicative of that sort of problem, but it could also be explained by delegating work to the CPU. Maybe both are related. I'll do some testing here.

I hope these directions convert into some real updates soon! Thanks a lot for your patience!

[FahimF]

Hey @pcuenca, thanks for taking a look and for the detailed notes — much appreciated 😄

• Didn't even know that there was a 1.13 pre-RC for PyTorch. I did notice that the nightly dev versions changed to 1.14 and that there was no 1.13 released but didn't realize what it signified. Will hunt down the RC and test with it. I haven't switched to Ventura yet, but since it will release next week, it shouldn't be long ...
• Thank you for the link to performance fixes on the PyTorch side. I've been looking for any indications of performance fixes and was following several threads but hadn't seen that one. Looks interesting!
• Regarding memory size, two things — 1. I have 32GB of RAM on my machine 2. A larger size image (768 x 512) completes in 58 seconds with diffusers 0.3.0 but takes almost 20 minutes with 0.5.1. So would memory be the cause there? Would it not seem unlikely since it works fine with the older version of diffusers?

Thank you again for your update, the testing, and all that you do, and if you need me to test anything at any point or provide further info, please do let me know.

[pcuenca]

Hey @FahimF, small update. I tested diffusers 0.3.0 and this branch: #926, both on PyTorch "pre RC", which you can install like so:

pip install --pre torch --extra-index-url https://download.pytorch.org/whl/test/cpu

(It uses test instead of nightly in the URL).

These are my results:

Width	Diffusers	Time
768	0.3.0	101.47
	m1-torch-1.13	83.02
512	0.3.0	44.80
	m1-torch-1.13	39.95

I'm using a small warmup of 10 steps in both cases. This is the code I'm testing with:

from diffusers import StableDiffusionPipeline
import time
import torch

device = "mps"
seed = 1337

sdm = StableDiffusionPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
)
sdm = sdm.to(device)
    
prompt = "A painting of a squirrel eating a burger"

# Warmup
torch.manual_seed(seed)
images = sdm(prompt, num_inference_steps=10).images

# Actual inference
torch.manual_seed(seed)
start = time.time()
images = sdm(prompt).images
print(f"[{device}] - {time.time()-start:.2f}")

It'd be really interesting to know if you still see that dramatic difference you described when running at a size larger than 512. Please, let me know if you have any questions or insights :) Thanks a lot!

[FahimF]

Hey @pcuenca, will do the testing either tonight or tomorrow morning using your script for all variations - without PyTorch 13 pre-RC, with it, and for the two different image sizes and let you know what the figures are 😄

[pcuenca]

Awesome, thanks a lot! 😄

[FahimF]

@pcuenca here are the results:

macOS version is 12.6

RAM on device: 32GB

The height is 512 for all the following tests and the Python version is 3.9.13 and the warm up run always did 512 x 512 images but for 10 steps.

PyTorch versions used are 1.13.0.dev20220924 (before PyTorch image generation slowed down) and PyTorch 1.13.0 (pre RC).

Installed 1.13.0.dev20220924 using: pip install --upgrade --pre torch==1.13.0.dev20220924 --extra-index-url https://download.pytorch.org/whl/nightly/cpu

Installed 1.13.0 using: pip install --pre torch --extra-index-url https://download.pytorch.org/whl/test/cpu

Configuration	width = 512	width = 768
torch 1.13.0.dev20220924 diffusers 0.3.0	Warm up - 10s Full - 32.02s	Warm up - 13s 56.57s
torch 1.13.0.dev20220924 diffusers 0.6.0	Warm up - 13s Full - 41s	Warm up - 12s Full - 36m 45s
torch 1.13.0 diffusers 0.3.0	Warm up - 1m 40s Full - 6m 57s	Warm up - 1m 37s Full - 30m 16s
torch 1.13.0 diffusers 0.6.0	Warm up - 14s Full - 58s	Warm up - 14s Full - 7m 37s

As you'll note, the best times were for the older PyTorch 1.13.0 dev release and diffusers 0.3.0. The latest diffusers and the PyTorch pre RC does seem to improve timing but there's a weird outlier in terms of generation time for diffusers 0.3.0 and the 1.13.0 pre RC for 512 images. I ran most of these last night before went to bed. If you need me to re-run any of them, please let me know ...

[pcuenca]

Thanks a lot @FahimF, really appreciated! I'll try to get a M1 machine with macOS 12.6 to see what impact the OS has.