indices should be either on cpu or on the same device as the indexed tensor (cpu)

[Vortitron]

Describe the bug

image_to_image.py line 92 throws the error above.

  `  init_latents = self.scheduler.add_noise(init_latents, noise, timesteps)`

I've tried adding .to(self.device) to the 3 parameters.
Device should be 'cuda' though.

Reproduction

device = "cuda"

pipei2i = StableDiffusionImg2ImgPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
    revision="fp16", 
    torch_dtype=torch.float16,
    use_auth_token=True
).to(device)

response = requests.get('https://pbs.twimg.com/media/Fa1_7_vWYAEwfX-.png')

init_image = Image.open(io.BytesIO(response.content)).convert("RGB")
init_image = init_image.resize((512, 512))
init_image = preprocess(init_image)

outputs = pipei2i(prompt=prompt, init_image=init_image, strength=0.75, num_inference_steps=75,guidance_scale=0.75)

Logs

No response

System Info

diffusers==0.2.4
nvidia/cuda:11.3.0-cudnn8-devel-ubuntu20.04

[patrickvonplaten]

@patil-suraj could you take a look here?

[Vortitron]

I downloaded the source and tracked the problem down to this line...
diffusers/schedulers/scheduling_pndm.py:254

    sqrt_alpha_prod = self.alphas_cumprod[timesteps] ** 0.5

[Vortitron]

Solved:
Maybe I should listen to the error message sometime.. it should all be cpu
init_latents = self.scheduler.add_noise(init_latents, noise, timesteps)
becomes
init_latents = self.scheduler.add_noise(init_latents, noise, timesteps.cpu())
and all is well.

[loretoparisi]

Solved:

Maybe I should listen to the error message sometime.. it should all be cpu

init_latents = self.scheduler.add_noise(init_latents, noise, timesteps)

becomes

init_latents = self.scheduler.add_noise(init_latents, noise, timesteps.cpu())

and all is well.

Is .to(device) not supported for timesteps then?

[Vortitron]

Solved:
Maybe I should listen to the error message sometime.. it should all be cpu
init_latents = self.scheduler.add_noise(init_latents, noise, timesteps)
becomes
init_latents = self.scheduler.add_noise(init_latents, noise, timesteps.cpu())
and all is well.

Is .to(device) not supported for timesteps then?

I think it was supported but 'device' was cuda, and it's trying to do something with numpy which uses cpu

[patil-suraj]

Hi @andydhancock

This is working fine for me. Here's the code snippet I tried.

cd into examples/inference and run

import requests
from PIL import Image
from io import BytesIO

from torch import autocast


from image_to_image import *

device = "cuda"
pipei2i = StableDiffusionImg2ImgPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
    revision="fp16", 
    torch_dtype=torch.float16,
    use_auth_token=True
).to(device)


response = requests.get('https://pbs.twimg.com/media/Fa1_7_vWYAEwfX-.png')
init_image = Image.open(BytesIO(response.content)).convert("RGB")
init_image = init_image.resize((512, 512))
init_image = preprocess(init_image)

prompt = "a cat, artstation"

with autocast("cuda"):
    outputs = pipei2i(prompt=prompt, init_image=init_image, strength=0.75, num_inference_steps=75,guidance_scale=0.75)

[Vortitron]

Hi @andydhancock

This is working fine for me. Here's the code snippet I tried.

cd into examples/inference and run

import requests
from PIL import Image
from io import BytesIO

from torch import autocast


from image_to_image import *

device = "cuda"
pipei2i = StableDiffusionImg2ImgPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
    revision="fp16", 
    torch_dtype=torch.float16,
    use_auth_token=True
).to(device)


response = requests.get('https://pbs.twimg.com/media/Fa1_7_vWYAEwfX-.png')
init_image = Image.open(BytesIO(response.content)).convert("RGB")
init_image = init_image.resize((512, 512))
init_image = preprocess(init_image)

prompt = "a cat, artstation"

with autocast("cuda"):
    outputs = pipei2i(prompt=prompt, init_image=init_image, strength=0.75, num_inference_steps=75,guidance_scale=0.75)

Strange it wouldn't on mine then! I got it to work with the timesteps.cpu() thing anyways so not a problem anymore.

[loretoparisi]

@andydhancock thanks! On Apple M1 it worked in cpu removing FP16 and autocast:
🥇

import requests
from PIL import Image
from io import BytesIO

from torch import autocast

from image_to_image import *

device = 'cuda' if torch.cuda.is_available() else 'cpu'
print(f'running on {device}')

pipei2i = StableDiffusionImg2ImgPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
    #revision="fp16", 
    #torch_dtype=torch.float16,
    use_auth_token=True
).to(device)


response = requests.get('https://pbs.twimg.com/media/Fa1_7_vWYAEwfX-.png')
init_image = Image.open(BytesIO(response.content)).convert("RGB")
init_image = init_image.resize((512, 512))
init_image = preprocess(init_image)

prompt = "a cat, artstation"
samples = 1
outputs = []
if device=='cuda':
    with autocast("cuda"):
        outputs = pipei2i(prompt=[prompt]*samples, 
            init_image=init_image, 
            #strength=0.75, 
            num_inference_steps=50,
            guidance_scale=7.5)
else:
    outputs = pipei2i(prompt=[prompt]*samples,
        init_image=init_image, 
        #strength=0.75, 
        num_inference_steps=50,
        guidance_scale=7.5)

A question, the default value guidance_scale was 0.75 not 7.5 as above, right? And same for strength I assume.
Thank you.

[Vortitron]

Yeah, guidance_scale is 7.5, strength 0.75.. typo.. they're variables in my actual code

Wait why was this closed? Shouldn't this be fixed in source? Or is the Collab someone else's problem?

[patrickvonplaten]

Doesn't this work for you @cherrerajobs:

import requests
from PIL import Image
from io import BytesIO

from torch import autocast


from diffusers import StableDiffusionImg2ImgPipeline

device = "cuda"
pipei2i = StableDiffusionImg2ImgPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
    revision="fp16", 
    torch_dtype=torch.float16,
    use_auth_token=True
).to(device)


response = requests.get('https://pbs.twimg.com/media/Fa1_7_vWYAEwfX-.png')
init_image = Image.open(BytesIO(response.content)).convert("RGB")
init_image = init_image.resize((512, 512))

prompt = "a cat, artstation"

with autocast("cuda"):
    outputs = pipei2i(prompt=prompt, init_image=init_image, strength=0.75, num_inference_steps=75,guidance_scale=0.75)

It should work now on master

[FahimF]

Using the code from main branch and the following code, it fails on Apple Silicon devices:

import torch
import requests
from PIL import Image
from io import BytesIO

from torch import autocast


from diffusers import StableDiffusionImg2ImgPipeline

device = torch.device("cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu")
pipei2i = StableDiffusionImg2ImgPipeline.from_pretrained(
    "stable-diffusion-v1-4",
).to(device)


response = requests.get('https://pbs.twimg.com/media/Fa1_7_vWYAEwfX-.png')
init_image = Image.open(BytesIO(response.content)).convert("RGB")
init_image = init_image.resize((512, 512))

prompt = "a cat, artstation"

with autocast("cpu"):
    outputs = pipei2i(prompt=prompt, init_image=init_image, strength=0.75, num_inference_steps=75,guidance_scale=0.75)

The error is as follows:

Traceback (most recent call last):
  File "test.py", line 24, in <module>
    outputs = pipei2i(prompt=prompt, init_image=init_image, strength=0.75, num_inference_steps=75,guidance_scale=0.75)
  File "/Users/fahim/miniforge3/envs/ml/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/Users/fahim/miniforge3/envs/ml/lib/python3.8/site-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_img2img.py", line 101, in __call__
    init_latents = self.scheduler.add_noise(init_latents, noise, timesteps)
  File "/Users/fahim/miniforge3/envs/ml/lib/python3.8/site-packages/diffusers/schedulers/scheduling_pndm.py", line 270, in add_noise
    sqrt_alpha_prod = self.alphas_cumprod[timesteps] ** 0.5
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)

It appears that self.alphas is on cpu while timesteps is on mps in scheduling_pndm.py on line 268? (I know it says 270 but I made some changes to the code there to check something ...)

If I change references to timesteps to timesteps.cpu() in that method, then the code works. But wouldn't it be better to put self.alphas_cumprod on whatever device is being used and so basically put that on the current device instead? Just asking since I'm not sure if that is a better idea or not ...

[patrickvonplaten]

Interesting! I don't have a mac computer - @patil-suraj @pcuenca do you have one? Could you give it a try maybe? :-)