-
Notifications
You must be signed in to change notification settings - Fork 6.1k
Description
We use monkey-patching to support LoRA fine-tuning of the text encoder and loading of the LoRA params corresponding to the text encoder. While it helps us in preventing additional dependencies like peft
*, it comes with baggage.
As pointed out by @rvorias in #3490 (comment), if we call pipe.load_lora_weights()
twice (with a checkpoint containing text encoder LoRA parameters), monkey-patching becomes recursive.
This should be prevented at all costs.
@rvorias also proposed a fix in takuma104#1 which tackles the issue by introducing a flag text_encoder_modify_forwards
which controls if LoRA for text encoder should progress or not. Once LoRA is applied for text encoder one-time, this flag is set to False.
If one needs to swap the text encoder back to its original state they could do:
del pipe.text_encoder
Then,
text_encoder = CLIPTextModel(ckpt_id, subfolder="text_encoder", torch_dtype=pipe.torch_dtype).to(pipe.device)
pipe.text_encoder = text_encoder
I would prefer this approach.
Calling @patrickvonplaten @pcuenca @takuma104 @rvorias to chime in too :)