Support for adding multiple LoRA layers to Diffusers

[kirit93]

Given the popularity of LoRA training for Stable Diffusion, it would be great if the support for adding LoRAs to Diffusers could be expanded to support adding multiple LoRAs. This would be an expansion of this feature - #2129 (comment).

[patrickvonplaten]

Agree, we should try to allow loading multiple LoRA layers - will put this on my TODO List.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[pustar]

thank you for your great work! expecting the support for adding multiple loras to diffuser

[patrickvonplaten]

cc @sayakpaul here as well

[yuananf]

What is the status of this issue now? Can we add multiple Loras for inference?

[sayakpaul]

Nope. Not yet. We're concretizing the interoperability with Kohya-style LoRA checkpoints along with support for SDXL too.

#4287

[yuananf]

Nope. Not yet. We're concretizing the interoperability with Kohya-style LoRA checkpoints along with support for SDXL too.

#4287

Thank you for your quick reply.

[gadicc]

And also, just a short note that we'll also need a way to adjust the scale (e.g. cross_attention_kwargs={"scale": 0.5}) on a per-LoRA basis 🙈