Support of the LyCORIS (LoCon/LoHA) models

[mkhennoussi]

Model/Pipeline/Scheduler description

Hi everyone ! Thanks for your amazing work !

Some specific (optimized) version of Lora are developed (https://github.com/KohakuBlueleaf/LyCORIS) and are available (https://civitai.com/models/37053/michael-jordan) with pretty cool features (ability to make distinctions between trained concepts, etc.). It could be nice, as support for loading .safetensors is made, to have the possibility to load also LyCORIS models.
What do you think ?

Thanks a lot !

Open source status

• The model implementation is available
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

No response

[JemiloII]

I would love to see this feature as well. However, most implementations will bake it into their LoRA implementations. So maybe extending LoRAs to use these and not check metadata names vigorously would be a start.

[JemiloII]

#3294 there is a pr here to support it.

[carl10086]

I encountered an issue while using version 0.17.1 of the library, specifically when calling the load_lora_weights method. I received the following error:

• it's a lycoris from this url: https://civitai.com/models/76404/lycoris-couture-by-edg

ValueError("Network alpha is not consistent")
This error originates from the following method:

_convert_kohya_lora_to_diffusers(state_dict)

To bypass this error, I tried commenting out the following lines of code:

    for key, value in state_dict.items():
        if "lora_down" in key:
            lora_name = key.split(".")[0]
            lora_name_up = lora_name + ".lora_up.weight"
            lora_name_alpha = lora_name + ".alpha"
            if lora_name_alpha in state_dict:
                alpha = state_dict[lora_name_alpha].item()
                if network_alpha is None:
                    network_alpha = alpha
                # elif network_alpha != alpha:
                #     raise ValueError("Network alpha is not consistent")

The code now runs, but this is not the appropriate solution.

I would appreciate any suggestions or advice on how to properly handle this

[patrickvonplaten]

cc @sayakpaul

[sayakpaul]

I encountered an issue while using version 0.17.1 of the library, specifically when calling the load_lora_weights method. I received the following error:

• it's a lycoris from this url: https://civitai.com/models/76404/lycoris-couture-by-edg

ValueError("Network alpha is not consistent")
This error originates from the following method:

_convert_kohya_lora_to_diffusers(state_dict)

To bypass this error, I tried commenting out the following lines of code:

    for key, value in state_dict.items():
        if "lora_down" in key:
            lora_name = key.split(".")[0]
            lora_name_up = lora_name + ".lora_up.weight"
            lora_name_alpha = lora_name + ".alpha"
            if lora_name_alpha in state_dict:
                alpha = state_dict[lora_name_alpha].item()
                if network_alpha is None:
                    network_alpha = alpha
                # elif network_alpha != alpha:
                #     raise ValueError("Network alpha is not consistent")

The code now runs, but this is not the appropriate solution.

I would appreciate any suggestions or advice on how to properly handle this

Did you get the expected outputs for this? We added that check for robustness in the module. Cc: @takuma104.

There can be different configurations for LyCORIS and from the get-go, it's not possible to support all of them. So, we started supporting them minimally: https://huggingface.co/docs/diffusers/main/en/training/lora#supporting-a1111-themed-lora-checkpoints-from-diffusers.

Question for @takuma104:

Do we want to relax this constraint?

diffusers/src/diffusers/loaders.py

Line 1242 in 0bab447

elif network_alpha != alpha:

I think that might break things as that would mean different alphas for different LoRA layers, no?

[carl10086]

1. I am in full agreement with your current viewpoint, which is why I have chosen to raise this issue in this particular thread concerning LyCORIS.

2. LyCORIS in a1111 does have an array of formats, making it indeed challenging to support them all at once.

3. LyCORIS getting more and more in civitai :)

[takuma104]

@sayakpaul The constraint you mention indeed assumes that all network_alphas are same, as you stated. This assumption comes from the simplification of propagating network_alpha through a single variable. To correct this, we could potentially propagate all network_alphas that correspond to each lora-weight. I believe we should incorporate this improvement when we address the LoCon revisions. From what I've gathered, some LoCon files have unique network_alphas, while others do not.

I believe the LoCon support essentially extends the methodology of #3756. Therefore, I think it would be best to first finalize #3756 as the mechanism, and then create a separate PR for LoCon support.

[sayakpaul]

I concur with your thoughts, @takuma104. #3778 is about to be get merged soon. So, we can start #3756 and what you have gathered pretty soon.

Cc: @patrickvonplaten

[sayakpaul]

@takuma104 went through https://gist.github.com/takuma104/dcf4626fe2b0564d02c6edd4e9fcb616 I saw either 4 or 32 for the LoRAs you have listed there. But within the same LoRA, the alpha value didn't change. Perhaps, I missed something?

[takuma104]

@sayakpaul The first comment is mostly with network_alpha=4, but there are some parts where network_alpha=1. This key is unfamiliar to me, so I think this might be the LoCon part.

lora_unet_down_blocks_0_downsamplers_0_conv.alpha 1.0 1 1.0
lora_unet_down_blocks_0_resnets_0_conv1.alpha 1.0 1 1.0
lora_unet_down_blocks_0_resnets_0_conv2.alpha 1.0 1 1.0
lora_unet_down_blocks_0_resnets_1_conv1.alpha 1.0 1 1.0
lora_unet_down_blocks_0_resnets_1_conv2.alpha 1.0 1 1.0

[sayakpaul]

@carl10086 we will first have support for the rest of blocks as mentioned in #3087 (comment). Then we will revisit this as this concerns a change in how we deal with network_alphas in the LoRA layers.

Feel free to bug us in the coming weeks :)