How could I convert a LoRA .safetensors or .ckpt file into the format that diffusers can process?

[garyhxfang]

Describe the bug

I got some LoRA model in .safetensors format, and tried to convert in to the format that can be used in diffusers.
But nowhere I can find any document or scripts to achieve that.
So I try to convert the file with the convert_original_stable_diffusion_to_diffusers.py scripts, but it didn't work.
Could somebody provide a guideline or script about how should I covert the LoRAs?

Reproduction

python convert_original_stable_diffusion_to_diffusers.py --checkpoint_path /xxx/yyy/zzz.safetensors --scheduler_type euler-ancestral --dump_path /aaa/bbb/ccc --from_safetensors

and i got the following error

───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /root/imagine/notebook/convert.py:105 in │
│ │
│ 102 │ parser.add_argument("--device", type=str, help="Device to use (e.g │
│ 103 │ args = parser.parse_args() │
│ 104 │ │
│ ❱ 105 │ pipe = load_pipeline_from_original_stable_diffusion_ckpt( │
│ 106 │ │ checkpoint_path=args.checkpoint_path, │
│ 107 │ │ original_config_file=args.original_config_file, │
│ 108 │ │ image_size=args.image_size, │
│ │
│ /root/miniconda3/lib/python3.8/site-packages/diffusers/pipelines/stable_diff │
│ usion/convert_from_ckpt.py:945 in │
│ load_pipeline_from_original_stable_diffusion_ckpt │
│ │
│ 942 │ unet_config["upcast_attention"] = upcast_attention │
│ 943 │ unet = UNet2DConditionModel(**unet_config) │
│ 944 │ │
│ ❱ 945 │ converted_unet_checkpoint = convert_ldm_unet_checkpoint( │
│ 946 │ │ checkpoint, unet_config, path=checkpoint_path, extract_ema=ex │
│ 947 │ ) │
│ 948 │
│ │
│ /root/miniconda3/lib/python3.8/site-packages/diffusers/pipelines/stable_diff │
│ usion/convert_from_ckpt.py:335 in convert_ldm_unet_checkpoint │
│ │
│ 332 │ │
│ 333 │ new_checkpoint = {} │
│ 334 │ │
│ ❱ 335 │ new_checkpoint["time_embedding.linear_1.weight"] = unet_state_dic │
│ 336 │ new_checkpoint["time_embedding.linear_1.bias"] = unet_state_dict[ │
│ 337 │ new_checkpoint["time_embedding.linear_2.weight"] = unet_state_dic │
│ 338 │ new_checkpoint["time_embedding.linear_2.bias"] = unet_state_dict[ │
╰──────────────────────────────────────────────────────────────────────────────╯
KeyError: 'time_embed.0.weight'

Logs

No response

System Info

diffusers 0.11.0 python3.8

[jndietz]

I would pull down main locally, and run pip install -e . inside that directory. That will give you the latest pre-released version of diffusers. Upgrading both diffusers and transformers resolved some issues for me in regards to going from safetensors/ckpt -> diffusers in the last day.

[garyhxfang]

@jndietz Thanks, but do you know is convert_original_stable_diffusion_to_diffusers.py this script the correct way to convert the LoRA file from *.safetensors to diffusers format?

I though it cannot be used to convert the LoRA file but as i cannot find any other scipts related, I give it a try and failed

[jndietz]

@garyhxfang Based on this:

https://huggingface.co/spaces/diffusers/sd-to-diffusers/blob/main/app.py

It seems like, yes, convert_original_stable_diffusion_to_diffusers.py would be the correct script. However, since this is a LoRA, I'm not sure if that is the correct approach. Are LoRAs merged with a checkpoint the same as a LoRA on its own? I'm not sure. It's worth a shot. You'll have to provide something for --original_config_file, which perhaps the original SD 1.5 yaml will work? All hypothetical - it's worth trying.

[jndietz]

Looking at the app.py script further, I think you would have to convert from safetensors, to pytorch(ckpt), then to diffusers. I'm starting to wonder what the value in huggingface's diffusers library is... I feel like I've spent more time messing around with moving formats than actually using it for creation the last week or so.

[garyhxfang]

@jndietz Thanks, let me have a try again.

[garyhxfang]

Still cannot convert the safetensors LoRA file, could master @patrickvonplaten or @patil-suraj help to have a look on this issue?
As LoRA becomes more and more popular recently, while the communities get used to share LoRA file in safetensors or ckpt format, I think it's quite important that we have an convenient approach/script to convert the LoRA file to something diffusers can process.

Also I found that we don't have solution on convert/load the textual inverse embedding in .pt format, I spent all they searching for the solution but doesn't find any approach.

[patrickvonplaten]

Hey @garyhxfang ,

Note that the conversion script you used won't work as it expects a full model checkpoint. Could you upload your LoRA embeddings somewhere so that we could write a new conversion script?

How did you get the LoRA checkpoints for this? Which notebook did you use for training?

[garyhxfang]

Thanks a lot for your response! @patrickvonplaten

I download the LoRA file from civitai.com, you can also have a check on the website, it is one of the most important websites/communities that people share their stable diffusion and LoRA models. I think many of them train the LoRA model with AUTOMATIC111/stable-diffusion-webui.

The LoRA model I would like to convert are
https://civitai.com/models/7448/korean-doll-likeness
and
https://civitai.com/models/7716/taiwan-doll-likeness
I also upload it to hf
https://huggingface.co/garyfang/LoRA_Collection/tree/main

Since many many popular LoRA models are shared in .safetensors format, and the end users love LoRA very much as it really perform very well in generating great images, it's quite important for us developers that diffusers could have a generic approach/script to convert the .safetensors LoRA models into the format can be loaded by diffusers. So that developers using diffusers could make use of these great models already in the communities to build better models or applications.
Currently the next release of my application really need these LoRAs to be embedded to the model, it will be so great if we could get support from your side to achieve that.

Thanks very much.

[patrickvonplaten]

BTW this seems related: #2326

[dkackman]

it's quite important for us developers that diffusers could have a generic approach/script to convert the .safetensors LoRA models > into the format can be loaded by diffusers.

Better yet, support .safetensors LoRA models directly

[Skquark]

I ran into this too trying to load or convert LatentLabs360 safetensors LoRA file so I can use it with the new Panorama pipeline, but no matter what I tried in the convert script or loading pretrained, I couldn't get it to work. Model is here https://civitai.com/models/10753/latentlabs360 if anyone wants to try... I now see that convert_lora_safetensor_to_diffusers.py is being worked on, so I guess I can wait for that to get merged.

[sunhaha123]

I ran into this too trying to load or convert LatentLabs360 safetensors LoRA file so I can use it with the new Panorama pipeline, but no matter what I tried in the convert script or loading pretrained, I couldn't get it to work. Model is here https://civitai.com/models/10753/latentlabs360 if anyone wants to try... I now see that convert_lora_safetensor_to_diffusers.py is being worked on, so I guess I can wait for that to get merged.

look this: https://github.com/kohya-ss/sd-scripts/blob/main/networks/merge_lora_old.py