ValueError: Attempting to unscale FP16 gradients. in diffusers/examples/research_projects/lora/train_text_to_image_lora.py

[liushanyuan18]

Describe the bug

Thanks for this excellent work！
When I run diffusers/examples/research_projects/lora/train_text_to_image_lora.py, I encountered this issue:
ValueError: Attempting to unscale FP16 gradients

Could you please tell me how can I fix it?

Reproduction

accelerate launch train_text_to_image_lora.py
--pretrained_model_name_or_path=$MODEL_NAME
--train_data_dir=$DATASET_NAME
--caption_column="text"
--resolution=512 --random_flip
--train_batch_size=8
--mixed_precision="fp16"
--gradient_accumulation_steps=4
--num_train_epochs=200 --checkpointing_steps=500
--learning_rate=1e-04 --lr_scheduler="constant" --lr_warmup_steps=0
--seed=42
--output_dir=$OUTPUT_DIR
--validation_prompt=''
--report_to="wandb"
--use_peft
--lora_r=4 --lora_alpha=32
--lora_text_encoder_r=4 --lora_text_encoder_alpha=32
--enable_xformers_memory_efficient_attention

Logs

No response

System Info

• diffusers version: 0.15.0.dev0
• Platform: Linux-5.4.0-42-generic-x86_64-with-glibc2.10
• Python version: 3.8.5
• PyTorch version (GPU?): 1.13.1+cu117 (True)
• Huggingface_hub version: 0.13.3
• Transformers version: 4.25.1
• Accelerate version: 0.16.0
• xFormers version: 0.0.16rc424
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

[patrickvonplaten]

Gently pinging https://github.com/haofanwang here

[haofanwang]

@liushanyuan18

It should be related to this issue.

(1) Try to remove mixed_precision=fp16 in your command, and use fp32 by default.
(2) If (1) works, you can then try comment our following lines. This seems to be typos. For training modules, they are required to be in fp32.

unet.to(accelerator.device, dtype=weight_dtype)
text_encoder.to(accelerator.device, dtype=weight_dtype)

Fix it in #2795. Besides, you should set --train_text_encoder if you want to train lora for text encoder.

[viiika]

@liushanyuan18

It should be related to this issue.

(1) Try to remove mixed_precision=fp16 in your command, and use fp32 by default. (2) If (1) works, you can then try comment our following lines. This seems to be typos. For training modules, they are required to be in fp32.

unet.to(accelerator.device, dtype=weight_dtype)
text_encoder.to(accelerator.device, dtype=weight_dtype)

Fix it in #2795. Besides, you should set --train_text_encoder if you want to train lora for text encoder.

Great solution! We should not set training modules' dtype to weight_dtype. It solves my another problem related to ValueError: Attempting to unscale FP16 gradients