Skip to content

StableDiffusionKDiffusionPipeline does not support LoRA loading #3750

@tripathiarpan20

Description

@tripathiarpan20

LoRA loading is supported in the StableDiffusionPipeline thanks to the last release (v0.17.0), however it's not yet fully supported in other pipelines like StableDiffusionKDiffusionPipeline, this is probably under work anyway given this PR , but I'm adding a PR anyway as it is a really simple feature to add by simply changing

class StableDiffusionKDiffusionPipeline(DiffusionPipeline, TextualInversionLoaderMixin):

to

class StableDiffusionKDiffusionPipeline(DiffusionPipeline, TextualInversionLoaderMixin, LoraLoaderMixin):

I have also tested the same with my branch with the following code on a LoRA checkpoint (trained using kohya-ss trainer):

pipe = StableDiffusionKDiffusionPipeline.from_pretrained('/content/lora/training_scripts/stable-diffusion-v1-5', 
    torch_dtype=torch.float16,
    safety_checker = None,
).to('cuda')
pipe.set_scheduler('sample_dpmpp_2m')
pipe.load_lora_weights(".", weight_name="</content/topdowobliquefurniture.safetensors")
pipe._lora_scale = 0.8
prompt = "stone castle, topdowobliquefurniture"
negative_prompt = ("(low quality, worst quality:1.4), (grainy:2.0)")

images = pipe(prompt=prompt, 
    negative_prompt=negative_prompt, 
    width=512, 
    height=512,
    guidance_scale = 7.0, 
    num_inference_steps=15, 
    num_images_per_prompt=1,
    use_karras_sigmas=True
    # generator=torch.manual_seed(0)
).images
images[0]

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions