Batch Processing for Multi ControlNet.

[ahmadmustafaanis]

Describe the bug

I am trying to use Batch Processing for Multi ControlNet, but it does not work as intended.

Reproduction

pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "my_model", torch_dtype=torch.float16,
    controlnet=[
        controlnet_pose, 
        controlnet_hed
    ],
).to("cuda")

output = pipe(
        ["prompt 1", "prompt 2"],   # Batch Processing for 2 prompts
        [[pose_image_low, hed_image_low], [pose_image_cc, hed_image_cc]],  # 2 Images for each prompt
        generator = [torch.Generator(device="cuda").manual_seed(16605685601386031741), torch.Generator(device="cuda").manual_seed(16605685601386031741)],  # 2 generators for each prompt
        negative_prompt=["low quality, wrong face, distorted face, ugly, beard", "low quality, wrong face, distorted face, tikka, ugly, beard"],  # 2 neg prompts
        num_inference_steps=50,
    controlnet_conditioning_scale = [1, 1.2, 1, 0.7]  # This gives an error
controlnet_conditioning_scale = [[1, 1.2], [1, 0.7]] # this gives an error
controlnet_conditioning_scale = [1, 1.2] # this works fine but I want to use different conditioning scale for both prompts
).images

Logs

No response

System Info

Latest installation

pip install git+https://github.com/huggingface/diffusers

GPU: Tesla T4

[patrickvonplaten]

Yes this looks like an error to me, thanks for the issue @ahmadmustafaanis . @williamberman could you have a look here?

[takuma104]

At present, the controlnet_conditioning_scale parameter does not support batches. It did not allow list before Multi-ControlNet support was added. The current version with Multi-ControlNet support still inherits this specification and does not support batches. If controlnet_conditioning_scale = [1, 1.2] is specified, it does not apply to batches, but rather corresponds to each ControlNet. controlnet_conditioning_scale=1 is passed to controlnet_pose, and controlnet_conditioning_scale=1.2 is passed to controlnet_hed.

It can be confusing to determine which parameter supports batches and which does not, as both Batch and Multi-ControlNet use list specification. Is there any better solution to this problem?

[takuma104]

        [[pose_image_low, hed_image_low], [pose_image_cc, hed_image_cc]],  # 2 Images for each prompt

This part is also not very straightforward, but in the current implementation, it is necessary to do it this way to achieve the intended behavior.

        [[pose_image_low, pose_image_cc], [hed_image_low, hed_image_cc]],  # 2 Images for each prompt

[patrickvonplaten]

@williamberman @sayakpaul could you check here?

[sayakpaul]

It can be confusing to determine which parameter supports batches and which does not, as both Batch and Multi-ControlNet use list specification. Is there any better solution to this problem?

I agree. I think we could just log information to the users when these cases arrive to keep the API design not too complicated. One reference for that would be how it's handled in the InstructPix2Pix pipeline:

diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_instruct_pix2pix.py

Line 730 in 7fe8861

f"You have passed {batch_size} text prompts (`prompt`), but only {image_latents.shape[0]} initial"

[patrickvonplaten]

Let's try to open a PR here to fix it maybe @sayakpaul or @williamberman ?

[sayakpaul]

Closing this issue in favor of #2855