diffusers 0.30.0 StableDiffusionXLControlNetPipeline dtype error

[yagmurx]

Describe the bug

Description:

I am defining two ControlNetModel instances using the diffusers library and creating a pipeline using StableDiffusionXLControlNetPipeline. When I try to generate with this pipeline, I encounter an error in diffusers version 0.30.0, although it works fine in version 0.27.2. The error message is:

RuntimeError: mat1 and mat2 must have the same dtype, but got Float and Half

Additional Information:

• The issue is present in diffusers version 0.30.0 but not in version 0.27.2.
• The models and pipeline are all moved to CUDA.

Reproduction

from diffusers import (
     StableDiffusionXLPipeline,
     ControlNetModel,
     StableDiffusionXLControlNetPipeline
     )
     
SDXL_MODEL_PATH = "/local/path/to/safetensors/"
SDXL_MODEL_CONF =  "/local/path/to/config_yaml/"

pipe = StableDiffusionXLPipeline.from_single_file(
    SDXL_MODEL_PATH,
    original_config_file=SDXL_MODEL_CONF,
    torch_dtype=torch.float16,
    text_encoder=text_encoder,
    text_encoder_2=text_encoder_2,
    tokenizer=tokenizer,
    tokenizer_2=tokenizer,
    local_files_only=True,
    use_safetensors=True,
    add_watermarker=False,
    feature_extractor=None,
).to("cuda")

controlnets = [
    ControlNetModel.from_pretrained(
        "/local/path/to/destitech/controlnet-inpaint-dreamer-sdxl/",
        torch_dtype=torch.float16,
#         variant="fp16",
    ).to("cuda"),
    ControlNetModel.from_pretrained(
        "/local/path/to/diffusers/controlnet-zoe-depth-sdxl-1.0/",
        torch_dtype=torch.float16,
    ).to("cuda"),
]
pipeline = StableDiffusionXLControlNetPipeline(**pipe.components, controlnet=controlnets)


image = pipeline(
                   prompt,
                   negative_prompt=negative_prompt,
                   image= [init_image, zoe_mask],
                   guidance_scale=6.5,
                   num_inference_steps=25,
                   generator=generator,
                   controlnet_conditioning_scale=[0.5, 0.8],
                   control_guidance_end=[0.9, 0.6],
                   ).images[0]
                   
 image.save("output.jpg")
````

### Logs

_No response_

### System Info

- 🤗 Diffusers version: 0.30.0
- Platform: Linux-4.18.0-372.107.1.el8_6.x86_64-x86_64-with-glibc2.35
- Running on Google Colab?: No
- Python version: 3.10.12
- PyTorch version (GPU?): 2.3.1+cu121 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Huggingface_hub version: 0.23.3
- Transformers version: 4.41.2
- Accelerate version: 0.30.1
- PEFT version: not installed
- Bitsandbytes version: not installed
- Safetensors version: 0.4.3
- xFormers version: not installed
- Accelerator: (secret, 32GB)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@sayakpaul 

[tolgacangoz]

Hi @yagmurx. There are many unknown elements; could you provide them to make it fully reproducible? Also, full error trace, please (in order to spot the exact location).
Could you double-check that text_encoder and text_encoder_2 are in torch.float16?

[andrzejpiasecki]

I encountered the same error, although using different pipelines.

File "/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/controlnet/pipeline_controlnet_sd_xl.py", line 1500, in __call__
  noise_pred = self.unet(

File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1552, in _wrapped_call_impl
  return self._call_impl(*args, **kwargs)

File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1561, in _call_impl
  return forward_call(*args, **kwargs)

File "/app/src/controlnet/kohya_hires_fix.py", line 225, in forward
  encoder_hidden_states = self.process_encoder_hidden_states(

File "/usr/local/lib/python3.10/dist-packages/diffusers/models/unets/unet_2d_condition.py", line 1036, in process_encoder_hidden_states
  image_embeds = self.encoder_hid_proj(image_embeds)

File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1552, in _wrapped_call_impl
  return self._call_impl(*args, **kwargs)

File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1561, in _call_impl
  return forward_call(*args, **kwargs)

File "/usr/local/lib/python3.10/dist-packages/diffusers/models/embeddings.py", line 1715, in forward
  image_embed = image_projection_layer(image_embed)

File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1552, in _wrapped_call_impl
  return self._call_impl(*args, **kwargs)

File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1561, in _call_impl
  return forward_call(*args, **kwargs)

File "/usr/local/lib/python3.10/dist-packages/diffusers/models/embeddings.py", line 1598, in forward
  x = self.proj_in(x)

File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1552, in _wrapped_call_impl
  return self._call_impl(*args, **kwargs)

File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1561, in _call_impl
  return forward_call(*args, **kwargs)

File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/linear.py", line 116, in forward
  return F.linear(input, self.weight, self.bias)

RuntimeError: mat1 and mat2 must have the same dtype, but got Float and Half

[asomoza]

if you use with the exact same code, but with a diffusers model with from_pretrained instead of from_single_file does the error still happen?

[yiyixuxu]

@andrzejpiasecki @yagmurx can either of you share an reproducible script (including the inputs and checkpoints) - that will help a lot for us to understand the problem!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[yagmurx]

Okay, here is my solution. I got this error because of text_encoder's and tokenizer's dtype.

I got the error with this:

from transformers import CLIPTextModel, CLIPTextModelWithProjection, CLIPTokenizer
clip_path = "some/local/path"

text_encoder = CLIPTextModel.from_pretrained(
                       clip_path,
                       local_files_only=True,
                       )

I've add a torch_dtype input to this func, and I solve the error.

from transformers import CLIPTextModel, CLIPTextModelWithProjection, CLIPTokenizer
clip_path = "some/local/path"

text_encoder = CLIPTextModel.from_pretrained(
                       clip_path,
                       torch_dtype=torch.float16,
                       local_files_only=True,
                       )

In the first code I posted, I create all encoders and tokenizers with the local path like this. I changed the dtype in all of their codes.

[nokommo]

File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/linear.py", line 116, in forward
return F.linear(input, self.weight, self.bias)

RuntimeError: mat1 and mat2 must have the same dtype, but got Float and Half

def forward(self, input: Tensor) -> Tensor:
    if input.dtype != self.weight.dtype:
        input = input.to(self.weight.dtype)
    return F.linear(input, self.weight, self.bias)