Add unload_ip_adapter method #6192
Conversation
src/diffusers/loaders/ip_adapter.py
Outdated
| self.config.encoder_hid_dim_type = None | ||
|
|
||
| # restore original Unet layers | ||
| device = self.unet.device |
There was a problem hiding this comment.
I was thinking, we just have to set the attention processor back to default here instead of reloading the UNet again.
cc @sayakpaul here, let me know what you think!
There was a problem hiding this comment.
Indeed that should be the way to go. We can first store the original attention processors and wap them back in during the unloading process.
Here's a reference that might be relevant:
src/diffusers/loaders/ip_adapter.py
Outdated
| ``` | ||
| """ | ||
| # remove CLIP image encoder | ||
| if hasattr(self, "image_encoder"): |
There was a problem hiding this comment.
Might be better:
if hasattr(self, "image_encoder") and getattr(self, "image_encoder", None) is not None:
sayakpaul
left a comment
sayakpaul
left a comment
There was a problem hiding this comment.
Thanks for taking the lead on this!
Have left some comments and I agree with @yiyixuxu that https://github.com/huggingface/diffusers/pull/6192/files#r1428359297 should be the way to do it.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
src/diffusers/loaders/ip_adapter.py
Outdated
| self.config.encoder_hid_dim_type = None | ||
|
|
||
| # restore original Unet attention processors layers | ||
| self.unet._restore_attn_processors() |
sayakpaul
left a comment
sayakpaul
left a comment
There was a problem hiding this comment.
Thanks for the changes! Could we add some tests here?
patrickvonplaten
left a comment
patrickvonplaten
left a comment
There was a problem hiding this comment.
Looks very nice! Let's maybe just remove the original_attn_processors variable for now to not introduce a new logic
src/diffusers/loaders/unet.py
Outdated
| # store original attn_processors if not already stored | ||
| if ( | ||
| getattr(self, "original_attn_processors", None) is None | ||
| or getattr(self, "original_attn_processors", None) == {} | ||
| ): | ||
| self.original_attn_processors = self.attn_processors |
There was a problem hiding this comment.
| # store original attn_processors if not already stored | |
| if ( | |
| getattr(self, "original_attn_processors", None) is None | |
| or getattr(self, "original_attn_processors", None) == {} | |
| ): | |
| self.original_attn_processors = self.attn_processors |
Can we maybe not add this mechanism for now? When unloading the IP adapters let's just set the default SDPA (torch 2.0) attention processors. This introduces too much magic for now IMO
|
@sayakpaul @yiyixuxu in the end I followed @patrickvonplaten's directions on restoring the default attention processors Added one simple test |
| if isinstance(attn_processor, (IPAdapterAttnProcessor, IPAdapterAttnProcessor2_0)): | ||
| attn_processor.scale = scale | ||
|
|
||
| def unload_ip_adapter(self): |
|
Great job @fabiorigano |
|
i think there may be sideeffects of using this - it removes any cross-attention applied to unet
e.g., if unet was previously using sdpa, after unload it will not. |
yes, attention processors with torch sdpa are not reloaded. should this maybe become
|
* Add unload_ip_adapter method * Update attn_processors with original layers * Add test * Use set_default_attn_processor --------- Co-authored-by: Sayak Paul <[email protected]>
6 participants
What does this PR do?
Fixes #6180
Who can review?
@yiyixuxu
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.