Min-SNR weight for v-prediction in examples

[thuliu-yt16]

In #5238, the Min-SNR weight for v-prediction is implemented as
$$w_t = \frac {min(\text{SNR}+1, \gamma)} {\text{SNR}+1}$$
but it should be
$$w_t = \frac {min(\text{SNR}, \gamma)} {\text{SNR}+1}$$.
See https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image.py#L931-L937.

[sayakpaul]

Cc: @bghira

[bghira]

can you submit a fix with some validation examples. i've done about 100,000 steps of tuning on the existing code. if something works better with that change, i'd love to see

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[bghira]

@thuliu-yt16 i would like for this to not be lost if the improvement can be quantified

[thuliu-yt16]

Sorry for the late reply. Which dataset would you suggest to validate the modification on? Or could you tell me which dataset #5238 is using if it's convenient?

[bghira]

sure, i host several public datasets on huggingface hub.

the model in 5238 was trained using a superset of mj-general and there is a new photorealism dataset of 10k high quality high-res images for you to do what you wish with.

[thuliu-yt16]

I tried to fine-tune stable diffusion 2-1 on the pseudo-images-10k dataset. I tried both the resolutions of 512 and 1024. The differences between SNR+1 and SNR are hard to tell. I guess it is partly due to the limited scale of data. As I don't have a lot of computing resources. Maybe I will leave this issue open. But mathematically, min(SNR, gamma) / (SNR + 1) for v-prediction is equivalent to min(SNR, gamma) for x0-prediction.

[bghira]

i have been doing my own testing and while it doesn't change how the model really learns eg. fine details, it does augment the loss values in ways that we might appreciate.

without your change the loss seems to floor to a value around 0.28-0.34 but with the change, the loss goes all the way to the expected value of 0.08-0.12 when converging.

if you want to open that PR, it sounds like a good idea to me

[thuliu-yt16]

In terms of the loss value, I think they can not be compared in that way as they are not calculated in the same way. Anyway, I have made a PR #6307. Hope this helps.

[bghira]

thanks for the PR. the loss value is passed into the backward method unaltered so in that regard they are comparable for the same dataset and training seed

[thuliu-yt16]

I mean both the losses are weighted but the weights are different. The SNR+1 version is always larger than the SNR version.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[bghira]

:|