[LoRA] add: test to check if peft loras are loadable in non-peft envs.

[sayakpaul]

What does this PR do?

Adds a test to ensure LoRA trained using peft (emulating how it's done in our trainers) can be fully loaded without peft.

Helps to keep our sanity in check :D

The dummy components were obtained using this Colab:
https://huggingface.co/datasets/diffusers/notebooks/blob/main/check_logits_with_serialization_peft_lora.py. This script also spits out the logits which are tested in this PR.

The basic idea is the following.

If we do:

sd_pipeline = DiffusionPipeline.from_pretrained(...)

torch.manual_seed(0)
unet_lora_config = LoraConfig(...)

torch.manual_seed(0)
text_encoder_lora_config = LoraConfig(...)

sd_pipeline.unet.add_adapter(unet_lora_config)
sd_pipeline.text_encoder.add_adapter(unet_lora_config)

outputs = sd_pipeline(**pipeline_inputs, generator=torch.manual_seed(0), output_type="np").images

And then if you do (with peft uninstalled):

sd_pipeline = DiffusionPipeline.from_pretrained(...)

# LoRA was obtained using what's shown in the script linked above. 
sd_pipeline.load_lora_weights(...)
outputs_no_peft = sd_pipeline(**pipeline_inputs, generator=torch.manual_seed(0), output_type="np").images

outputs and outputs_no_peft should match. This is exactly what we test here.

Cc: @apolinario

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[younesbelkada]

Thanks! Would it makes sense to test logits as well? that way we can make sure in the future no PR will break inference correctness. Here if I got it right we only test weights loading + dummy inference but not inference correctness - but this is already great so feel free to merge it I would say

[sayakpaul]

@younesbelkada just did. PTAL.

[younesbelkada]

Awesome work, thanks a lot @sayakpaul ! 🚀

[sayakpaul]

@younesbelkada feel free to look at the script with which I generated the LoRA files: https://huggingface.co/datasets/diffusers/notebooks/blob/main/check_logits_with_serialization_peft_lora.py.

I think they emulate how it's done in the training scripts. But I'd appreciate another set of 👁️s.

[DN6]

I think we might need a check to ensure this doesn't run in the PEFT environment as well right?

[sayakpaul]

Why can't it run on peft environment? Could you elaborate?

[DN6]

Why run the test in both the PEFT and non-PEFT environment/runners if it is only meant to check that we can load PEFT LoRAs in a non-PEFT environment?

[sayakpaul]

Oh okay. I think we're covering on that front:

diffusers/.github/workflows/pr_tests.yml

Line 100 in 1754602

python -m pytest -n 2 --max-worker-restart=0 --dist=loadfile \

doesn't run PEFT LoRA stuff.

All tests from https://github.com/huggingface/diffusers/blob/main/tests/lora/test_lora_layers_peft.p (which is tested in this workflow), always require the PEFT backend:

diffusers/tests/lora/test_lora_layers_peft.py

Line 78 in 1754602

@require_peft_backend

Does this make sense?

[patrickvonplaten]

Good idea!