ip-adapter-full-face_sd15 is producing corrupt / broken faces

[jroessler]

Describe the bug

I'm trying to use the the following IP Adapter using diffusers (ip-adapter-full-face_sd15.bin):
https://huggingface.co/h94/IP-Adapter/blob/main/models/ip-adapter-full-face_sd15.bin

However, as soon as I provide a human face as a reference image, the results look broken (see example below).

When using a different IP Adapter (e.g., ip-adapter_sd15.bin) or when using a different reference image (e.g., the one used in HF's tutorial here), everything is working.

I also tried installing different diffusers versions (i.e., 0.26, 0.28, 0.28.2).

Reproduction

from diffusers import AutoPipelineForText2Image
import torch
from diffusers.utils import load_image
from PIL import Image
from torchvision import transforms

pipeline = AutoPipelineForText2Image.from_pretrained("runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16).to("cuda")
pipeline.load_ip_adapter("h94/IP-Adapter", subfolder="models", weight_name="ip-adapter-full-face_sd15.bin")

generator = torch.Generator(device="cpu").manual_seed(33)
image = Image.open("input.png")

transform = transforms.Compose([
    transforms.Resize(512, interpolation=transforms.InterpolationMode.BILINEAR),
    transforms.CenterCrop(512),
])

images = pipeline(prompt="best quality, high quality, wearing sunglasses", ip_adapter_image=image, negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality", num_inference_steps=50, generator=generator).images[0]
images

Logs

No response

System Info

• diffusers version: 0.28.0.dev0
• Platform: Linux-5.15.0-107-generic-x86_64-with-glibc2.35
• Python version: 3.10.12
• PyTorch version (GPU?): 2.3.1+cu121 (True)
• Huggingface_hub version: 0.23.3
• Transformers version: 4.41.2
• Accelerate version: 0.31.0
• xFormers version: not installed
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: No

Who can help?

@yiyixuxu

[fabiorigano]

hi @jroessler it is not a bug, we noticed this issue when we integrated the model into the library (#5911 (comment))

You should set a different noise scheduler as explained in the documentation https://huggingface.co/docs/diffusers/main/en/using-diffusers/ip_adapter#face-model

[jroessler]

Hi @fabiorigano thank you so much! DDIM scheduler works like a charm :) Maybe we should change the wording from "recommended" to "mandatory" (or similar) in the documentation? Thanks again!