Loading cached pipeline takes too long

[patrickvonplaten]

Describe the bug

When loading a pipeline that is cached lot of time is wasted by calling the Hub unnecessarily. We should try to avoid this by trying to call the Hub only once if the model is cached.

See internal slack thread for more context.

Reproduction

Run the following script:

#!/usr/bin/env python3
from diffusers import StableDiffusionPipeline
import time
import sys

local_file_only = bool(int(sys.argv[1]))

start_time = time.time()
pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", local_files_only=local_file_only)
print(f"Local files only {str(local_file_only)}: {time.time() - start_time}")

with the args 0 and 1, one gets:

Local files only False: 2.1148853302001953
Local files only True: 0.31396007537841797

=> We do too many calls to the Hub when everything is already cached. We should only do one call to the Hub to check that everything is cached ideally.

Logs

No response

System Info

Main

[patrickvonplaten]

For comparison transformers only calls the Hub once

from transformers import BertModel
import time
import sys

local_file_only = bool(int(sys.argv[1]))

start_time = time.time()
model = BertModel.from_pretrained("bert-base-cased", local_files_only=local_file_only)
print(f"Local files only {str(local_file_only)}: {time.time() - start_time}")

gives

Local files only False: 0.943523645401001 
Local files only True: 0.5369486808776855 

[patrickvonplaten]

We should see how we can speed this up cc @Wauplin @williamberman @pcuenca

[Wauplin]

(thanks for opening the issue @patrickvonplaten. I updated your description by adding a link to the slack thread)

Here is what's happening when calling diffusers .from_pretrained(..., local_files_only=False):

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): huggingface.co:443
DEBUG:urllib3.connectionpool:[https://huggingface.co:443](https://huggingface.co/) "HEAD /CompVis/stable-diffusion-v1-4/resolve/main/model_index.json HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): huggingface.co:443
DEBUG:urllib3.connectionpool:[https://huggingface.co:443](https://huggingface.co/) "GET /api/models/CompVis/stable-diffusion-v1-4 HTTP/1.1" 200 72113
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): huggingface.co:443
DEBUG:urllib3.connectionpool:[https://huggingface.co:443](https://huggingface.co/) "GET /api/models/CompVis/stable-diffusion-v1-4/revision/main HTTP/1.1" 200 72113
Fetching 16 files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:00<00:00, 91304.58it/s]

So we have minimum 3 calls when all the files are cached:

• HEAD seems to check latest commit hash to download model_index.json => let's remember the commit_hash from this call (using get_hf_file_metadata)
• GET => this call checks if repo has safetensors or not => we can get rid of this by checking the cached files instead
• GET => this call lists files when using snapshot_download without local_files_only => we can improve if we know before-hand which files we want to have and if those are cached.

In the end we would reduce the number of calls to 1 in case files are already cached. We cannot get rid of this one since we want to check if the model has not been updated between 2 re-runs.

[wd255]

Local files only False: 2.1148853302001953
Local files only True: 0.31396007537841797

Curious about the environment where you get this number. I ran the same script on a colab T4, model loading time is two magnitudes higher

Edit - My bad. My nproc is 2. Jealous!

[laxuscullen]

Local files only False: 2.1148853302001953
Local files only True: 0.31396007537841797

Curious about the environment where you get this number. I ran the same script on a colab T4, model loading time is two magnitudes higher

Edit - My bad. My nproc is 2. Jealous!

Hey, did you figure out a way to speed things up? I am using local files true and it loads from cache always but it goes to around ~40s sometimes. I checked my nproc output, it says 4. I am using a tesla t4 with 15360MiB of memory.

[patrickvonplaten]

Ah it should be much faster now with: #2515

[laxuscullen]

Ah it should be much faster now with: #2515

I have asked for help multiple times now, even opened my own issue. Not sure how else to seek help regarding such a major issue I am facing? Tried pinging you as well but no help at all.

[patrickvonplaten]

Answered on the issue - I cannot reproduce your problem. Would be good to see a google colab that clearly shows that loading time is at 40 seconds

[laxuscullen]

Answered on the issue - I cannot reproduce your problem. Would be good to see a google colab that clearly shows that loading time is at 40 seconds

I have 40s of loading time including 4s of image generation. There’s no progress bar or anything shown when initiating the model using local files true. It just stays hanging until it’s loaded. The same is the issue locally as well but it’s a bit faster on mps as compared to cuda on the server. I am not sure what else to show you? It takes that much time that’s why I am talking about it. I have mentioned before as well I am not using Google collab. It’s aws server instance using Tesla T4 and 16gb ram.

[laxuscullen]

Answered on the issue - I cannot reproduce your problem. Would be good to see a google colab that clearly shows that loading time is at 40 seconds

Pretty bad support right here. Can you guide me towards where actual help will be provided regarding this issue?

[patrickvonplaten]

@laxuscullen as said before I cannot reproduce your issue and will have to see a reproducible code snippet. Google colab is the best option here.

[Wauplin]

@laxuscullen maybe something you can do is use cprofile to check which step is taking a lot of time. It is probable that there is a hardware bottleneck somewhere on your machine but it requires more information to debug it.

[laxuscullen]

@laxuscullen maybe something you can do is use cprofile to check which step is taking a lot of time. It is probable that there is a hardware bottleneck somewhere on your machine but it requires more information to debug it.

Hey man, you don't know how much I appreciate your reply. I was not aware of such a library existing! Here's what it says when sorted by total time descending:

@patrickvonplaten pinging you as well for help.

ncalls | tottime | percall | cumtime | percall | filename:lineno(function)
2122 | 34.056 | 0.016 | 34.084 | 0.016 | /home/api/venv/lib/python3.9/site-packages/torch/serialization.py:1076(load_tensor)
9162 | 4.047 | 0.000 | 4.074 | 0.000 | {method 'to' of 'torch._C._TensorBase' objects}
2 | 0.108 | 0.054 | 0.108 | 0.054 | {built-in method gc.collect}

Attaching the csv (as a zip) as well:
log_response.zip