Skip to content

The addition of 'load_ip_adapter' to the StableDiffusionXLAdapterPipeline #6167

@jquintanilla4

Description

@jquintanilla4

What API design would you like to have changed or added to the library? Why?
It would be great to have IP-adapters work with t2i-adapter as they're usually faster and less heavy on the image generation process than controlnet.

What use case would this enable or better enable? Can you give us a code example?

import torch
from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter, LCMScheduler
from diffusers.utils import load_image

model_id = 'stabilityai/stable-diffusion-xl-base-1.0'
lcm_lora_id = 'latent-consistency/lcm-lora-sdxl'
t2i_adapter_id = 'TencentARC/t2i-adapter-depth-midas-sdxl-1.0'

adapter = T2IAdapter.from_pretrained(t2i_adapter_id, torch_dtype=torch.float16, variant='fp16')

pipe = StableDiffusionXLAdapterPipeline.from_pretrained(model_id, adapter=adapter, torch_dtype=torch.float16, variant='fp16')
pipe.load_ip_adapter('h94/IP-Adapter', subfolder='sdxl_models', weight_name='ip-adapter_sdxl.safetensors')
pipe.load_lora_weights(lcm_lora_id)
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
pipe.enable_model_cpu_offload()

Current error:

AttributeError Traceback (most recent call last)
Cell In[6], line 12
9 adapter = T2IAdapter.from_pretrained(t2i_adapter_id, torch_dtype=torch.float16, variant='fp16').to('cuda')
11 pipe = StableDiffusionXLAdapterPipeline.from_pretrained(model_id, adapter=adapter, torch_dtype=torch.float16, variant='fp16')
---> 12 pipe.load_ip_adapter('h94/IP-Adapter', subfolder='sdxl_models', weight_name='ip-adapter_sdxl.safetensors')
13 pipe.load_ip_adapter('h94/IP-Adapter', subfolder='sdxl_models', weight_name='ip-adapter_sdxl.safetensors')
14 pipe.load_lora_weights(lcm_lora_id)

File ~/anaconda3/envs/diffusers2/lib/python3.9/site-packages/diffusers/configuration_utils.py:138, in ConfigMixin.getattr(self, name)
135 deprecate("direct config name access", "1.0.0", deprecation_message, standard_warn=False)
136 return self._internal_dict[name]
--> 138 raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")

AttributeError: 'StableDiffusionXLAdapterPipeline' object has no attribute 'load_ip_adapter'

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions