ValueError: For the given accelerator, there seems to be an unexpected problem in type-casting. Please file an issue on the PyTorch GitHub repository.

[christopher5106]

Describe the bug

Running the script examples/text_to_image/train_text_to_image_lora_sdxl.py
Not working anymore on mixed_precision fp16, bf16

Traceback (most recent call last):0%|██████████████████████████████████████████████████████████████| 7/7 [00:00<00:00, 67.30it/s]
File "train_text_to_image_lora_sdxl.py", line 1350, in
main(args)
File "train_text_to_image_lora_sdxl.py", line 1229, in main
images = [
File "train_text_to_image_lora_sdxl.py", line 1230, in
pipeline(**pipeline_args, generator=generator).images[0]
File "/home/ubuntu/360-training-preprocess/venv/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/ubuntu/360-training-preprocess/venv/lib/python3.8/site-packages/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py", line 1203, in call
raise ValueError(
ValueError: For the given accelerator, there seems to be an unexpected problem in type-casting. Please file an issue on the PyTorch GitHub repository. See also: https://github.com//pull/7446/.

Reproduction

To me, the regression comes from recent commit 544710e

#7446

I've never experiment this issue before.

Of course latents are in mixed precision before, and they are in float32 after passing through scheduler.

Logs

No response

System Info

Ubuntu, A10G, NVIDIA-SMI 550.54.14, Driver Version: 550.54.14, CUDA Version: 12.4
diffusers==0.28.0.dev0

Who can help?

@bghira @sayakpaul

[sayakpaul]

Can you share your training command?

[sayakpaul]

Cc: @bghira too

[bghira]

i don't actually have any CUDA devices to test this on. @sayakpaul is it possible we shouldn't be disabling autocast except on unsupported platforms?

[sayakpaul]

I think so yes.

[christopher5106]

Nothing special about my training command:

export MODEL_NAME="stabilityai/stable-diffusion-xl-base-1.0"
export VAE_NAME="madebyollin/sdxl-vae-fp16-fix"
export DATASET_DIR="../../../train"


accelerate launch train_text_to_image_lora_sdxl.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --pretrained_vae_model_name_or_path=$VAE_NAME \
  --train_data_dir=$DATASET_DIR --caption_column="text" \
  --resolution=1024 \
  --random_flip \
  --train_batch_size=1 --mixed_precision="fp16" \
  --num_train_epochs=2 --checkpointing_steps=500 \
  --learning_rate=1e-04 --lr_scheduler="constant" --lr_warmup_steps=0 \
  --seed=42 \
  --output_dir="sdxl" \
  --validation_prompt="cute dragon creatures" \
  --report_to="wandb"

I tried myself to force enable_autocast to True and False and this didn't change.

The origin of the problem is this comparison : https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py#L1198
During training, these dtype are always different.

[bghira]

well, try removing the exception test and see what happens next. on MPS, it has a broadcast error. maybe autocast fixes the broadcast errors on CUDA?

[christopher5106]

can you update the PR and I will test. Indeed there was an autocast somewhere because it was working fine before the error raise...

[bghira]

@christopher5106 that is pushed now.

[simbrams]

I'm running into the same issue with train_instruct_pix2pix_sdxl.py

GPU type: NVIDIA A100-SXM4-80GB

[bghira]

@simbrams i've updated that script in the same PR for you to test.

[simbrams]

Thanks @bghira!

[rfan-debug]

I can confirm i had the same issue on CUDA devices. My machine has a L4 GPU in this case. It seems that the commit fixes an issue for MPS but impacted CUDA devices unexpectedly.