Having a "scale" parameter for text encoder lora layers

[abhijitpal1247]

Is your feature request related to a problem? Please describe.
No, not related to a problem

Describe the solution you'd like
A similar parameter to "scale" of unet lora layers, in case of text encoder lora layers

Additional context
This is an inference parameter that will determine the influence of text encoder lora layers on generation

[patrickvonplaten]

cc @sayakpaul

[sayakpaul]

This question has come before a few times.

I am wondering about what would be the best option to support this currently. Here are some thoughts:

• Pass an additional parameter called text_encoder_scale to load_lora_weights() as a kwarg. Note that we cannot do it during the runtime (e.g., pipe("A pokemon with blue eyes.", cross_attention_kwargs={"scale": 0.5}) because of the way LoRA is implemented for the text encoder (using monkey-patching).
• If the text_encoder_scale is passed, then the monkey patch call will be modified maybe like so:

...
text_encoder_scale = kwargs.pop("text_encoder_scale", 1.0)

def new_forward(x):
    return old_forward(x) + lora_layer(x, scale=text_encoder_scale)

^ should be fine since LoRAAttnProcessor supports the scale argument in its call().

• Note that the current monkey-patching implementation has a bug which is fixed here: Fix monkey-patch for text_encoder LoRA #3490. Cc: @takuma104

WDYT @patrickvonplaten?

[patrickvonplaten]

Hmm yeah good question! I wonder whether we could do something ugly here:

diffusers/src/diffusers/loaders.py

Line 949 in d4197bf

def new_forward(x):

                def new_forward(x):
                    return old_forward(x) + lora_layer(x)

                # Monkey-patch.
                module.forward = new_forward

could be changed to

                def new_forward(x):
                    return old_forward(x) + self.scale * lora_layer(x)

                # Monkey-patch.
                module.forward = new_forward

and then we add a scale property to:

diffusers/src/diffusers/loaders.py

Line 745 in d4197bf

class LoraLoaderMixin:

@property
def scale(self):
    return self._scale

and a scale setter function that we need to call from the _encode_prompt method. Could this make sense?

[sayakpaul]

Where does self._scale come from? The pipeline call?

[patrickvonplaten]

Yes self._scale is overwritten when calling the pipeline

[sayakpaul]

@patrickvonplaten sorry still a bit unclear to me.

Yes self._scale is overwritten when calling the pipeline

To enable this we still need to expose an argument corresponding to the text encoder LoRA scale, no?

[patrickvonplaten]

@sayakpaul would this help: #3626 (comment)

[sayakpaul]

Thanks @patrickvonplaten! I left some comments there.

[sayakpaul]

Closing with #3626