Skip to content

[IP-Adapter] support face model #5886

@yiyixuxu

Description

@yiyixuxu

We recently added IP-adapter support to many of our pipelines in diffusers! You can now very easily load your IP-Adapter into a diffusers pipeline with pipe.load_ip_adapter(...);

However right now we do not support IPAdapterFull model checkpoints. Anyone interested to add that support for this? It will allow us to use the very cool face models with diffusers!

the goal would be to make this example work with diffusers API. e.g.

from diffusers import StableDiffusionPipeline
import torch
from diffusers.utils import load_image

base_model_path = "dreamlike-art/dreamlike-anime-1.0/"
pipeline = StableDiffusionPipeline.from_pretrained(
    base_model_path, torch_dtype=torch.float16)
pipeline.to("cuda")

image = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/ai_face2.png")

pipeline.load_ip_adapter("h94/IP-Adapter", subfolder="models", weight_name="ip-adapter-full-face_sd15.bin")
pipeline.set_ip_adapter_scale(0.6)
generator = torch.Generator(device="cpu").manual_seed(33)
images = pipeline(
    prompt="A photo of a girl wearing a black dress, holding red roses in hand, upper body, behind is the Eiffel Tower",
    ip_adapter_image=image,
    negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality", 
    num_inference_steps=50,
    generator=generator,
).images
images[0]

You will need to look into this PR to understand how load_ip_adapter works and then make it work for IPAdapterFull model from the original repo.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions