[Dreambooth Example] Attempting to unscale FP16 gradients.

[jpiabrantes]

Describe the bug

I had the training script working fine but then I updated diffusers to 0.7.2 and now I get the following error:

Traceback (most recent call last):
  File "/tmp/pycharm_project_990/train_dreambooth.py", line 938, in <module>
    main(args)
  File "/tmp/pycharm_project_990/train_dreambooth.py", line 876, in main
    optimizer.step()
  File "/opt/conda/envs/dreambooth/lib/python3.7/site-packages/accelerate/optimizer.py", line 134, in step
    self.scaler.step(self.optimizer, closure)
  File "/opt/conda/envs/dreambooth/lib/python3.7/site-packages/torch/cuda/amp/grad_scaler.py", line 337, in step
    self.unscale_(optimizer)
  File "/opt/conda/envs/dreambooth/lib/python3.7/site-packages/torch/cuda/amp/grad_scaler.py", line 282, in unscale_
    optimizer_state["found_inf_per_device"] = self._unscale_grads_(optimizer, inv_scale, found_inf, False)
  File "/opt/conda/envs/dreambooth/lib/python3.7/site-packages/torch/cuda/amp/grad_scaler.py", line 210, in _unscale_grads_
    raise ValueError("Attempting to unscale FP16 gradients.")
ValueError: Attempting to unscale FP16 gradients.
Steps:   0%|          | 0/800 [00:18<?, ?it/s]

Any ideas, or do I need to downgrade?

Reproduction

No response

Logs

No response

System Info

diffusers 0.7.2
python 3.7.12
accelerate 0.14.0

[jpiabrantes]

A bit more info. This happens on a fresh install when I set the --mixed_precision fp16 and --revision fp16

[gadicc]

Same here, but it's my first time trying to train.

@jpiabrantes I guess it will help if one of us can bisect this (i.e. find the exact commit where the bug was introduced) 😅 What were you using before 0.7.2? 0.7.1?

[cian0]

Same bug, I tried the fix here by changing the pytorch source directly and set allow fp16 = True , the training went through but the model only outputted black images

facebookresearch/fairscale#834

[gadicc]

As per the linked issue above, I think the actual issue seems to be in PyTorch?

In PyTorch/torch/cuda/amp/grad_scaler.py#L279 we have:

def unscale_(self, optimizer):
        # ...
        # The final False here is the allow_fp16=False argument to _unscale_grads_
        optimizer_state["found_inf_per_device"] = self._unscale_grads_(optimizer, inv_scale, found_inf, False)
        # ...

Here's are some relevant parts from _unscale_grads_:

    def _unscale_grads_(self, optimizer, inv_scale, found_inf, allow_fp16):
        # ...
                    if (not allow_fp16) and param.grad.dtype == torch.float16:
                        raise ValueError("Attempting to unscale FP16 gradients.")
        # ...

In pytorch/pytorch#74739 it's questioned why fp16 is disallowed.

@jpiabrantes (OP), since you had this working before, could you confirm not just the diffusers version you were using, but also PyTorch, and anything else you think might have changed? It would be really helpful if you could provide a known working configuration.

@cian0 thanks for mentioning that you already tried changing this value manually and it didn't work, saving us all some time! 🙏

[cian0]

To add, I also tried reverting the diffusers version, I am using Shivam's repo which is a fork of diffusers, he merged the latest version of diffusers which caused issues.

When I revert back to 15 days ago commit (when he hasn't merged yet), and downgrade my diffusers via pip, everything works again. Pytorch version never changed (v1.13.0) I also thought it was pytorch, I was supposed to downgrade pytorch if the revert solution didn't work but apparently I didn't have to.

[jpiabrantes]

I was using Shivam's repo as well with the fork of diffusers.

[gadicc]

Edit: the below can all be disregarded, much more progress from my next comment

Ok thanks @jpiabrantes. I'm able to reproduce consistently in pure diffusers. It's been excessively difficult to track down... I admit I'm quite new to Python but this all just seems crazy. I'll mention my findings so far, and maybe @patrickvonplaten can share some insight from his experience.

It seems the minute we switch from the 0.7.0.dev0 label to 0.7.0 we have this issue (and only changing the label, literally the commit that changes the label with no other code change), and then there's no going back again (you have to uninstall and reinstall a bunch of unrelated packages, otherwise commits that worked fine before no longer work). This will be clearer in these steps:

1. Create a fresh starting point

# Uninstalling all of these is the only way I can reliably "reset" the broken state
# Even though all of the versions stayed the same.
pip uninstall -r examples/dreambooth/requirements.txt
pip install accelerate torchvision ftfy

# Need this is a good starting point, otherwise other weird stuff is broken
pip install git+https://github.com/huggingface/diffusers@v0.6.0

2. Install the last commit that works (118c5be - "Docs: Do not require PyTorch nightlies")

$ pip install --no-cache git+https://github.com/huggingface/diffusers@118c5be94a2b8eb90fa41a2ceb59b3a8de9e0218
$ ./train # works!

pip log

$ pip install --no-cache git+https://github.com/huggingface/diffusers@118c5be94a2b8eb90fa41a2ceb59b3a8de9e0218
Defaulting to user installation because normal site-packages is not writeable
Collecting git+https://github.com/huggingface/diffusers@118c5be94a2b8eb90fa41a2ceb59b3a8de9e0218
  Cloning https://github.com/huggingface/diffusers (to revision 118c5be94a2b8eb90fa41a2ceb59b3a8de9e0218) to /tmp/pip-req-build-dg_2r41v
  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/diffusers /tmp/pip-req-build-dg_2r41v
  Running command git rev-parse -q --verify 'sha^118c5be94a2b8eb90fa41a2ceb59b3a8de9e0218'
  Running command git fetch -q https://github.com/huggingface/diffusers 118c5be94a2b8eb90fa41a2ceb59b3a8de9e0218
  Running command git checkout -q 118c5be94a2b8eb90fa41a2ceb59b3a8de9e0218
  Resolved https://github.com/huggingface/diffusers to commit 118c5be94a2b8eb90fa41a2ceb59b3a8de9e0218
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: regex!=2019.12.17 in /home/dragon/.local/lib/python3.10/site-packages (from diffusers==0.7.0.dev0) (2022.8.17)
Requirement already satisfied: filelock in /home/dragon/.local/lib/python3.10/site-packages (from diffusers==0.7.0.dev0) (3.8.0)
Requirement already satisfied: numpy in /usr/lib/python3.10/site-packages (from diffusers==0.7.0.dev0) (1.23.4)
Requirement already satisfied: huggingface-hub>=0.10.0 in /home/dragon/.local/lib/python3.10/site-packages (from diffusers==0.7.0.dev0) (0.11.0)
Requirement already satisfied: Pillow<10.0 in /usr/lib/python3.10/site-packages (from diffusers==0.7.0.dev0) (9.3.0)
Requirement already satisfied: requests in /home/dragon/.local/lib/python3.10/site-packages (from diffusers==0.7.0.dev0) (2.25.1)
Requirement already satisfied: importlib-metadata in /usr/lib/python3.10/site-packages (from diffusers==0.7.0.dev0) (5.0.0)
Requirement already satisfied: pyyaml>=5.1 in /usr/lib/python3.10/site-packages (from huggingface-hub>=0.10.0->diffusers==0.7.0.dev0) (6.0)
Requirement already satisfied: tqdm in /usr/lib/python3.10/site-packages (from huggingface-hub>=0.10.0->diffusers==0.7.0.dev0) (4.64.1)
Requirement already satisfied: packaging>=20.9 in /usr/lib/python3.10/site-packages (from huggingface-hub>=0.10.0->diffusers==0.7.0.dev0) (21.3)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/lib/python3.10/site-packages (from huggingface-hub>=0.10.0->diffusers==0.7.0.dev0) (4.4.0)
Requirement already satisfied: zipp>=0.5 in /usr/lib/python3.10/site-packages (from importlib-metadata->diffusers==0.7.0.dev0) (3.10.0)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3.10/site-packages (from requests->diffusers==0.7.0.dev0) (2022.9.24)
Requirement already satisfied: idna<3,>=2.5 in /home/dragon/.local/lib/python3.10/site-packages (from requests->diffusers==0.7.0.dev0) (2.10)
Requirement already satisfied: chardet<5,>=3.0.2 in /home/dragon/.local/lib/python3.10/site-packages (from requests->diffusers==0.7.0.dev0) (4.0.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.10/site-packages (from requests->diffusers==0.7.0.dev0) (1.26.12)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/lib/python3.10/site-packages (from packaging>=20.9->huggingface-hub>=0.10.0->diffusers==0.7.0.dev0) (3.0.9)
$ ./train

Steps:   0%|               | 1/400 [00:02<14:02,  2.11s/it, loss=0.135, lr=5e-6]╭───────────────────── Traceback (most recent call last) ──────────────────────╮
This should actually be considered WORKING because training starts
OutOfMemoryError: CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 15.75

GiB total capacity; 14.26 GiB already allocated; 222.56 MiB free; 14.49 GiB

reserved in total by PyTorch) If reserved memory is >> allocated memory try

setting max_split_size_mb to avoid fragmentation.  See documentation for Memory

Management and PYTORCH_CUDA_ALLOC_CONF

CalledProcessError: Command '['/usr/bin/python', 'train_dreambooth.py',

'--pretrained_model_name_or_path=CompVis/stable-diffusion-v1-4',

'--instance_data_dir=instance_images', '--output_dir=output_dir',

'--instance_prompt=a photo of sks dog', '--resolution=512',

'--train_batch_size=1', '--gradient_accumulation_steps=1',

'--learning_rate=5e-6', '--lr_scheduler=constant', '--lr_warmup_steps=0',

'--max_train_steps=400', '--revision=fp16', '--mixed_precision=fp16']' returned

non-zero exit status 1.

3. Install the first commit that breaks (1578679 - "Release: v0.7.0")

$ pip install -no-cache 1578679ff4a4ff8157214081438aa7d78f13b4fc
$ ./train # ValueError: Attempting to unscale FP16 gradients.

pip log

$ pip install --no-cache git+https://github.com/huggingface/diffusers@1578679ff4a4ff8157214081438aa7d78f13b4fc
Defaulting to user installation because normal site-packages is not writeable
Collecting git+https://github.com/huggingface/diffusers@1578679ff4a4ff8157214081438aa7d78f13b4fc
  Cloning https://github.com/huggingface/diffusers (to revision 1578679ff4a4ff8157214081438aa7d78f13b4fc) to /tmp/pip-req-build-ce_8p_6r
  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/diffusers /tmp/pip-req-build-ce_8p_6r
  Running command git rev-parse -q --verify 'sha^1578679ff4a4ff8157214081438aa7d78f13b4fc'
  Running command git fetch -q https://github.com/huggingface/diffusers 1578679ff4a4ff8157214081438aa7d78f13b4fc
  Running command git checkout -q 1578679ff4a4ff8157214081438aa7d78f13b4fc
  Resolved https://github.com/huggingface/diffusers to commit 1578679ff4a4ff8157214081438aa7d78f13b4fc
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: huggingface-hub>=0.10.0 in /home/dragon/.local/lib/python3.10/site-packages (from diffusers==0.7.0) (0.11.0)
Requirement already satisfied: regex!=2019.12.17 in /home/dragon/.local/lib/python3.10/site-packages (from diffusers==0.7.0) (2022.8.17)
Requirement already satisfied: filelock in /home/dragon/.local/lib/python3.10/site-packages (from diffusers==0.7.0) (3.8.0)
Requirement already satisfied: Pillow<10.0 in /usr/lib/python3.10/site-packages (from diffusers==0.7.0) (9.3.0)
Requirement already satisfied: importlib-metadata in /usr/lib/python3.10/site-packages (from diffusers==0.7.0) (5.0.0)
Requirement already satisfied: numpy in /usr/lib/python3.10/site-packages (from diffusers==0.7.0) (1.23.4)
Requirement already satisfied: requests in /home/dragon/.local/lib/python3.10/site-packages (from diffusers==0.7.0) (2.25.1)
Requirement already satisfied: packaging>=20.9 in /usr/lib/python3.10/site-packages (from huggingface-hub>=0.10.0->diffusers==0.7.0) (21.3)
Requirement already satisfied: tqdm in /usr/lib/python3.10/site-packages (from huggingface-hub>=0.10.0->diffusers==0.7.0) (4.64.1)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/lib/python3.10/site-packages (from huggingface-hub>=0.10.0->diffusers==0.7.0) (4.4.0)
Requirement already satisfied: pyyaml>=5.1 in /usr/lib/python3.10/site-packages (from huggingface-hub>=0.10.0->diffusers==0.7.0) (6.0)
Requirement already satisfied: zipp>=0.5 in /usr/lib/python3.10/site-packages (from importlib-metadata->diffusers==0.7.0) (3.10.0)
Requirement already satisfied: chardet<5,>=3.0.2 in /home/dragon/.local/lib/python3.10/site-packages (from requests->diffusers==0.7.0) (4.0.0)
Requirement already satisfied: idna<3,>=2.5 in /home/dragon/.local/lib/python3.10/site-packages (from requests->diffusers==0.7.0) (2.10)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3.10/site-packages (from requests->diffusers==0.7.0) (2022.9.24)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.10/site-packages (from requests->diffusers==0.7.0) (1.26.12)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/lib/python3.10/site-packages (from packaging>=20.9->huggingface-hub>=0.10.0->diffusers==0.7.0) (3.0.9)
Building wheels for collected packages: diffusers
  Building wheel for diffusers (pyproject.toml) ... done
  Created wheel for diffusers: filename=diffusers-0.7.0-py3-none-any.whl size=305127 sha256=6e9b12cc0ee68b4250af92ced689ef3d44aadbbc5fca57acf3355797da765f91
  Stored in directory: /tmp/pip-ephem-wheel-cache-lg6be8e4/wheels/d5/ad/09/71a9b17f6282e5cc00f53be606e4e230db6962308ae661308f
Successfully built diffusers
Installing collected packages: diffusers
  Attempting uninstall: diffusers
    Found existing installation: diffusers 0.7.0.dev0
    Uninstalling diffusers-0.7.0.dev0:
      Successfully uninstalled diffusers-0.7.0.dev0
Successfully installed diffusers-0.7.0
(base) [dragon@dragon d2]$ ./train
/home/dragon/.local/lib/python3.10/site-packages/accelerate/accelerator.py:205: UserWarning: `log_with=tensorboard` was passed but no supported trackers are currently installed.
  warnings.warn(f"`log_with={log_with}` was passed but no supported trackers are currently installed.")
{'weight_dtype': torch.float16}
Steps:   0%|                                            | 0/400 [00:00

After this (it breaking), you can literally go back to any previous commit that worked before - which all have the version 0.7.0.dev0 - and we still get the ValueError: Attempting to unscale FP16 gradients.. Going back to v0.6.0 works! But then going back to later commits with 0.7.0.dev0 still fails. Except if you do it in a weird funky order that I'm still figuring out, then it works (previous working commit fails, even earlier commits fails, but, go back to 0.6.0, now if you go back to one of the earlier commits, it works, and back to the previous working commit, which is now working again too 🤷‍♂️).

I thought maybe it was a deps issue but pip list before and after the problem arises remain identical (except for that first move from 0.7.0.dev0 to 0.7.0; but move back again and the deps match and the problem remains).

There really must be some weird Python thing going on here that's just beyond me... I could guess some file is being overwritten somewhere, and the fact that a lot of the different commits still have the same 0.7.0.dev0 version is confusing things... but pip --no-cache doesn't make any difference, even when it says it's creating a new wheel.

I hope I don't sound crazy, because working through all this definitely makes me feel crazy. I hope I've given enough info to hint to someone who knows the Python ecosystem more intimately to have an idea of what's going on. Happy to help out however else I can.

[patrickvonplaten]

@patil-suraj could you please take a look here?

[gadicc]

Ok I'm not sure what's up with my setup, but I had much better luck inside a docker container using git checkout directly. My sanity has returned and I've bisected the issue to this commit:

7482178 default fast model loading

🎉

[gadicc]

And indeed, passing fast_load=False to the unet loader (and only the unet) is enough to get this working (with slower loads of course). On the above commit at least. On later the latest main, I get the error again. Back to bisecting :D

[gadicc]

Ok, so, even with fast_load=False on the unet, this breaks again in:

42bb459 [Low cpu memory] Correct naming and improve default usage

Setting low_cpu_mem_usage=False on the unet fixes this one too, all the way up to the most recent commit on main:

So basically, currently possible to work around this issue with:

    unet = UNet2DConditionModel.from_pretrained(
        args.pretrained_model_name_or_path,
        subfolder="unet",
        revision=args.revision,
        # Add these two lines below to workaround the issue
        fast_load=False,
        low_cpu_mem_usage=False,
    )

Ok, that's it from me for the day... sorry for all the traffic. But I think this will be very helpful for @patil-suraj 😅 And has been a great make up experience for me personally after my last attempt 😅 😅

[patil-suraj]

Thanks for the detailed issue, taking a look now.