[IP-Adapter] add a 'unload_ip_adapter' method

[yiyixuxu]

fix #6088

Is anyone interested in adding a unload_ip_adapter method? it needs to undo everything we did with load_ip_adapter method:

diffusers/src/diffusers/loaders/ip_adapter.py

Line 46 in 93ea26f

def load_ip_adapter(

It needs to do below 3 things:

1. remove image_encoder
2. undo the changes we made to the UNet structure here (

   diffusers/src/diffusers/loaders/unet.py

   Line 82 in 93ea26f
   ), i.e.
   python self.encoder_hid_proj = None self.config.encoder_hid_dim_type = None
3. reset the attention processor

[fabiorigano]

Hi @yiyixuxu can I work on this? Thanks

[ArunMunagala7]

Can I work on this? Thanks

[yiyixuxu]

@fabiorigano of course!

[yiyixuxu]

Hi @ArunMunagala7

Thanks! @fabiorigano volunteered first, and I think it is a relatively easy issue that one contributor would be sufficient.
We still need a PR onunload_textual_inversion() if that interest you #6013