LoRA dreambooth example is broken

[AlexUmnov]

Describe the bug

Following the instructions from https://github.com/huggingface/diffusers/tree/main/examples/dreambooth do not yield personalised image.

I noticed the problem in my own script, but decided to try and follow the instruction before raising it and it didn't work also.
Looks like there was a recent commit to this folder

Reproduction

All the code can be found in the notebook https://colab.research.google.com/drive/1rloHRIqgWapF2dXn58ygjgduKzBZbOtW?usp=sharing

Logs

No response

System Info

diffusers @ https://github.com/huggingface/diffusers@4a3d528
transformers==4.35.2
Python 3.10.12

Who can help?

@yiyixuxu @DN6 @sayakpaul @patrickvonplaten

[sayakpaul]

This is probably some version problem. I am getting the expected results.

WandB: https://wandb.ai/s_t2i/dreambooth-lora/runs/fic3afzh

Colab Notebook: https://colab.research.google.com/gist/sayakpaul/11e9137801ce53fb0d83c0f3f0546ca7/scratchpad.ipynb

[AlexUmnov]

@sayakpaul thank you for a quick response. I cannot open your WandB link unfortunately.
Could you try to generate something with you model in the notebook?
Perhaps the problem is not in the training, but in loading the finetuned model.

[sayakpaul]

Here: https://wandb.ai/sayakpaul/dreambooth-lora/runs/50yk45c7.

If you look into the images tagged under "test", they are basically what you asked here:

Could you try to generate something with you model in the notebook?

[AlexUmnov]

Thanks @sayakpaul
I think the problem then is in the instructions on how to inference said trained model
Here's the new notebook https://colab.research.google.com/drive/1N2HT_kinvKRf9peLyWS8snVf6IJgKytv?usp=sharing

In the WandB run you can see that the predictions are good
https://wandb.ai/alexvumnov/dreambooth-lora/runs/hucxxi33/workspace?workspace=user-alexvumnov

But when I load it according to instructions:

import torch

pipe = DiffusionPipeline.from_pretrained(os.environ['MODEL_NAME'], torch_dtype=torch.float16)
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
pipe.to("cuda")
pipe.unet.load_attn_procs("/content/dreambooth_output")

The model doesn't seem updated

[AlexUmnov]

Update, I was able to fix it, we just need to call a different function to load the pipeline:

from diffusers import StableDiffusionPipeline
import torch

pipe = StableDiffusionPipeline.from_pretrained(os.environ['MODEL_NAME']).to("cuda")

pipe.load_lora_weights("/content/dreambooth_output"

Let me make a PR for the documentation change

[sayakpaul]

Alright. Closing it then.