Running a batch of prompts and negative prompts in SemanticStableDiffusionPipeline causes tensor dimension mismatch #4149
Description
Describe the bug
Hi,
I noticed that running a batch of prompts and negative prompts with the SemanticStableDiffusionPipeline causes tensor dimension mismatch between the negative prompt embeddings and prompt embeddings.
I believe a fix to this would be in Line 485 of diffusers/pipelines/semantic_stable_diffusion/pipeline_semantic_stable_diffusion.py: uncond_embeddings = uncond_embeddings.repeat(batch_size, num_images_per_prompt, 1) to uncond_embeddings = uncond_embeddings.repeat(1, num_images_per_prompt, 1). I tested this with a batch of prompts and negative prompts, and also for a single case.
Reproduction
from diffusers import SemanticStableDiffusionPipeline
import torch
import numpy as np
device="cuda:0" # Change to your device
pipe=SemanticStableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5").to(device)
seeds=np.random.choice(range(1,1000),10,replace=False)
generator= [torch.Generator(device=device).manual_seed(int(seed)) for seed in seeds]
negative_prompts="""original character, children, cartoon"""
out = pipe(prompt=["Any string"]*len(generator), generator=generator, guidance_scale=7.5,
num_inference_steps=50, negative_prompt= [negative_prompts]*len(generator),
editing_prompt=['sunglasses, wearing sunglasses']*len(generator),
reverse_editing_direction=[False]*len(generator),
edit_warmup_steps=[10]*len(generator),
edit_guidance_scale=[7.5]*len(generator),
edit_threshold=[0.99]*len(generator),
edit_momentum_scale=0.3,
edit_mom_beta=0.6,
)
Logs
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
Cell In[1], line 11
9 generator_i = [torch.Generator(device=device).manual_seed(int(seed)) for seed in seeds]
10 negative_prompts="""original character, children, cartoon"""
---> 11 out = pipe(prompt=["Any string"]*len(generator_i), generator=generator_i, guidance_scale=7.5,
12 num_inference_steps=50, negative_prompt= [negative_prompts]*len(generator_i),
13 editing_prompt=['sunglasses, wearing sunglasses']*len(generator_i),
14 reverse_editing_direction=[False]*len(generator_i),
15 edit_warmup_steps=[10]*len(generator_i),
16 edit_guidance_scale=[7.5]*len(generator_i),
17 edit_threshold=[0.99]*len(generator_i),
18 edit_momentum_scale=0.3,
19 edit_mom_beta=0.6,
20 )
File ~/anaconda3/lib/python3.10/site-packages/torch/utils/_contextlib.py:115, in context_decorator.<locals>.decorate_context(*args, **kwargs)
112 @functools.wraps(func)
113 def decorate_context(*args, **kwargs):
114 with ctx_factory():
--> 115 return func(*args, **kwargs)
File ~/anaconda3/lib/python3.10/site-packages/diffusers/pipelines/semantic_stable_diffusion/pipeline_semantic_stable_diffusion.py:494, in SemanticStableDiffusionPipeline.__call__(self, prompt, height, width, num_inference_steps, guidance_scale, negative_prompt, num_images_per_prompt, eta, generator, latents, output_type, return_dict, callback, callback_steps, editing_prompt, editing_prompt_embeddings, reverse_editing_direction, edit_guidance_scale, edit_warmup_steps, edit_cooldown_steps, edit_threshold, edit_momentum_scale, edit_mom_beta, edit_weights, sem_guidance)
492 if enable_edit_guidance:
493 print(uncond_embeddings.shape, text_embeddings.shape, edit_concepts.shape)
--> 494 text_embeddings = torch.cat([uncond_embeddings, text_embeddings, edit_concepts])
495 else:
496 text_embeddings = torch.cat([uncond_embeddings, text_embeddings])
RuntimeError: Sizes of tensors must match except in dimension 0. Expected size 7680 but got size 768 for tensor number 1 in the list.System Info
Python 3.10,
Diffusers 0.17.1
Who can help?
No response