Fix running LoRA with xformers

[bddppq]

#2247
#2124

from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-2-1")
pipe.unet.load_attn_procs(lora_dir)
pipe.to('cuda:2')

prompt = 'a cute cat'

image = pipe(prompt, guidance_scale=9, num_inference_steps=25).images[0]
image.save('1.png')

pipe.enable_xformers_memory_efficient_attention()
image = pipe(prompt, guidance_scale=9, num_inference_steps=25).images[0]
image.save('2.png')

pipe.disable_xformers_memory_efficient_attention()
image = pipe(prompt, guidance_scale=9, num_inference_steps=25).images[0]
image.save('3.png')

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[patrickvonplaten]

Great job @bddppq ! This looks exactly like it should :-)

[patrickvonplaten]

Could we maybe add one quick tests that ensures that the two implementations are identical. It should be pretty easy to add as this test:

1. Copy-paste this test:

   diffusers/tests/models/test_models_unet_2d_condition.py

   Line 388 in fd5c3c0

   def test_lora_on_off(self):

   and give it a new name with "xformers" in it
2. Verify that xformers on/off gives same results

That would be amazing!

Also happy to do it if you're too busy #1877

[patrickvonplaten]

Thanks a lot!

[juancopi81]

I found this issue while noticing that my inference times also increased after loading a LoRA. I am using a LoRA from civitai:

pipe.load_lora_weights("/path/to/lora/", weight_name = 'add_detail.safetensors')

Is it normal that now inference takes longer? Should I load the model differently? It is not clear for me from the example if I should first run the model without xformers and then activate it.

Thank you very much!

[juancopi81]

Nevermind! I found

pipe.fuse_lora()

and this worked!