The default learning rate of LCM LoRA training is too small

[Zeqiang-Lai]

Describe the bug

I am not sure if this script been tested, but it seems that the default lr (1e-6) is too small in my test. The model won't learn anything util I have increase it to 1e-4.

export MODEL_NAME="runwayml/stable-diffusion-v1-5"
export OUTPUT_DIR="path/to/saved/model"

accelerate launch train_lcm_distill_lora_sd_wds.py \
    --pretrained_teacher_model=$MODEL_NAME \
    --output_dir=$OUTPUT_DIR \
    --mixed_precision=fp16 \
    --resolution=512 \
    --lora_rank=64 \
    --learning_rate=1e-6 --loss_type="huber" --adam_weight_decay=0.0 \
    --max_train_steps=1000 \
    --max_train_samples=4000000 \
    --dataloader_num_workers=8 \
    --train_shards_path_or_url="pipe:curl -L -s https://huggingface.co/datasets/laion/conceptual-captions-12m-webdataset/resolve/main/data/{00000..01099}.tar?download=true" \
    --validation_steps=200 \
    --checkpointing_steps=200 --checkpoints_total_limit=10 \
    --train_batch_size=12 \
    --gradient_checkpointing --enable_xformers_memory_efficient_attention \
    --gradient_accumulation_steps=1 \
    --use_8bit_adam \
    --resume_from_checkpoint=latest \
    --report_to=wandb \
    --seed=453645634 \
    --push_to_hub \

Maybe we could lift the learning rate up for LoRA training or warn users to adjust the LR by themselves.

Reproduction

None

Logs

No response

System Info

None

Who can help?

No response

[dg845]

It looks like the default learning rate value is 1e-4 in the LCM LoRA SD script:

diffusers/examples/consistency_distillation/train_lcm_distill_lora_sd_wds.py

Lines 618 to 624 in df76a39

	# ----Learning Rate----
	parser.add_argument(
	"--learning_rate",
	type=float,
	default=1e-4,
	help="Initial learning rate (after the potential warmup period) to use.",
	)

and in the LCM LoRA SD-XL script:

diffusers/examples/consistency_distillation/train_lcm_distill_lora_sdxl_wds.py

Lines 599 to 604 in df76a39

	parser.add_argument(
	"--learning_rate",
	type=float,
	default=1e-4,
	help="Initial learning rate (after the potential warmup period) to use.",
	)

So I think it's just the example arguments in the READMEs that have a learning rate of 1e-6. I will open a PR to change the examples.

[rohit901]

Hi @dg845 do we also need to change the learning rate in the normal LCM script to 1e-4 from 1e-6, in the example script diffusers/examples/consistency_distillation/train_lcm_distill_sd_wds.py and in the corresponding readme.MD [diffusers/examples/consistency_distillation/README.md]?