IP Adapter Image Embeds - Compatibility with ComfyUI and other libraries/apps

[asomoza]

Since we now have this PR #6868 merged and as suggested in this comment #6868 (comment) I'm opening this issue to discuss the option to make the embeds for IP Adapters compatible with ComfyUI and other systems.

I've prepared a collab to demonstrate the issue right now:

https://colab.research.google.com/drive/1vMSxzM0bnOzkqB7KcOBtOx0MKUJSMNKm?usp=sharing

Basically ComfyUI, InvokeAI and probably all other apps do the image projection at the same time when encoding the images but diffusers does it later in the forward of the unet here:

diffusers/src/diffusers/models/unets/unet_2d_condition.py

Line 1077 in 17612de

image_embeds = self.encoder_hid_proj(image_embeds)

The solution will be that diffusers also do the image projection when encoding the images.

@yiyixuxu @sayakpaul is it possible? or maybe you have other possible solutions for this?

Also I found another issue while testing that would be good to discuss but maybe I should open another issue for it, what I found is that one of the key features of being able to pass the embeds is that we don't need to load the image encoder, but currently the pipelines force you to either pass the image encoder or try to load it automatically.

[sayakpaul]

I will let @yiyixuxu comment on the first half of the thread since Yiyi already suggested something in #6868.

what I found is that one of the key features of being able to pass the embeds is that we don't need to load the image encoder, but currently the pipelines force you to either pass the image encoder or try to load it automatically.

Well, I think that is needed by design if you look into the code snippet in #6868 (comment).

In the prepare_ip_adapter_image_embeds() utility there calls encode_image() which, in turn, relies on the image_encoder. This is why, after preparing the IP Adapter image embeddings, we unload it by calling pipeline.unload_ip_adapter(). This sets the image_encoder to None:

diffusers/src/diffusers/loaders/ip_adapter.py

Line 210 in 7c8cab3

self.image_encoder = None

I believe this is a clean enough solution that lets you do things transparently while also keeping the memory requirements at bay.

[yiyixuxu]

I think this can be supported with very minimum code changes

1. we allow passing the ip_hidden_state directly with cross_attention_kwargs argument from pipeline, basically, right here

   diffusers/src/diffusers/models/unets/unet_2d_condition.py

   Line 1071 in 7c8cab3

   elif self.encoder_hid_proj is not None and self.config.encoder_hid_dim_type == "ip_image_proj":

   we do something like

        elif self.encoder_hid_proj is not None and self.config.encoder_hid_dim_type == "ip_image_proj":
            if "ip_hidden_states" in cross_attention_kwargs:
                 ip_hidden_states = cross_attention_kwargs.pop("ip_hidden_states")

            elif "image_embeds" not in added_cond_kwargs:
                raise ValueError(
                    f"{self.__class__} has the config param `encoder_hid_dim_type` set to 'ip_image_proj' which requires the keyword argument `image_embeds` to be passed in  `added_conditions`"
                )
            else:
                image_embeds = added_cond_kwargs.get("image_embeds")
                ip_hidden_states = self.encoder_hid_proj(image_embeds)
            
             encoder_hidden_states = (encoder_hidden_states, ip_hidden_states)

2. in the load_ip_adapter function, we will make it possible to skip loading image_encoder. I think we can add a image_encoder_folder argument and default to "image_encoder" if you do this, it won't load image_encoder

pipe.load_ip_adapter("h94/IP-Adapter",  subfolder="sdxl_models", image_encoder_subfolder=None, weight_name=ip-adapter-plus_sdxl_vit-h.safetensors)

would this work for you? the obvious drawback for this is that this going to be a "hidden feature" that's not so easy to discover by users but I don't think it is a super big use case, no? we will of course document this well in our doc

[asomoza]

thank you both for your responses.

Lets first assume that I have the embeddings saved as a file, so the image encoder is not needed at all, the idea of the hidden argument to pass as a None would work for this perfect.

Using the cross_attention_kwargs to skip the ip_hidden_states = self.encoder_hid_proj(image_embeds) is what I use right now so it works perfect.

I agree though, this is probably not a big use case for diffusers, this would only be useful for:

• UIs that use diffusers as a backend
• People who do mass generations with IP Adapters
• People with low VRAM that can't encode the images in the same run as the generation.

tagging @cubiq as maybe he can provide insights if people use the loading and saving of the embeddings since the IP Adapter node is the only one I see that has that option.

also @vladmandic and @hipsterusername if they want to provide their opinions since their UIs use diffusers as a backend.

If this is really a low use case, feel free to ignore it since it’s easy for me to patch it on my end.

[yiyixuxu]

Hi @asomoza

We are very happy to support this because the code change is very small and I don't see any downside to adding this feature

I think what I said here might be a little bit confusing - what I meant it, by supporting this feature vis cross_attention_kwargs, it is not super great for people to discover this feature; But I don't think it matters much here

the obvious drawback for this is that this going to be a "hidden feature" that's not so easy to discover by users but I don't think it is a super big use case, no? we will of course document this well in our doc

[yiyixuxu]

@asomoza
if you are ok with the plan here #6925 (comment)
happy to open an PR

[sayakpaul]

I very much like the idea of supporting this. I think we can create a separate section in the IP Adapter guide doc that Steven is adding here: #6897.

This section could be "Memory optimization when using IP-Adapters" and there we document everything related to ip_adapter_image_embeds and ip_hidden_states. WDYT @yiyixuxu @asomoza?

[asomoza]

@asomoza if you are ok with the plan here #6925 (comment) happy to open an PR

This plan works perfect for me

This section could be "Memory optimization when using IP-Adapters" and there we document everything related to ip_adapter_image_embeds and ip_hidden_states. WDYT @yiyixuxu @asomoza?

looks good to me since is the main reason I'm using it for.

[cubiq]

@asomoza it's used mostly to spare resources with very (very) low spec GPUs. On comfyui saves you about 1gb of vram, I have no idea on diffusers.

[vladmandic]

IMO, borderline, but still useful use-case, thanks for looping me in.

[hipsterusername]

All seems to have been pretty well discussed.

Will note that I’m appreciative of willingness to incorporate things that allow for more flexibility for those implementing on top of diffusers, even if outside the implicit remit of the project!

[yiyixuxu]

@hipsterusername
our pleasure!