Skip to content

[LoRA] Repurcussions of using monkey-patching for text encoder LoRA #3621

@sayakpaul

Description

@sayakpaul

We use monkey-patching to support LoRA fine-tuning of the text encoder and loading of the LoRA params corresponding to the text encoder. While it helps us in preventing additional dependencies like peft*, it comes with baggage.

As pointed out by @rvorias in #3490 (comment), if we call pipe.load_lora_weights() twice (with a checkpoint containing text encoder LoRA parameters), monkey-patching becomes recursive.

This should be prevented at all costs.

@rvorias also proposed a fix in takuma104#1 which tackles the issue by introducing a flag text_encoder_modify_forwards which controls if LoRA for text encoder should progress or not. Once LoRA is applied for text encoder one-time, this flag is set to False.

If one needs to swap the text encoder back to its original state they could do:

del pipe.text_encoder

Then,

text_encoder = CLIPTextModel(ckpt_id, subfolder="text_encoder", torch_dtype=pipe.torch_dtype).to(pipe.device)

pipe.text_encoder = text_encoder

I would prefer this approach.

Calling @patrickvonplaten @pcuenca @takuma104 @rvorias to chime in too :)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions