[SD3] vae.config.shift_factor missing in dreambooth training examples

[bendanzzc]

Describe the bug

shift_factor missing in traning code:
https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth_lora_sd3.py#L1617,
but used in inference code:
https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion_3/pipeline_stable_diffusion_3.py#L893
Is it resonable that when traning SD3, we do not need to norm latents using vae.config.shift_factor and scale_factor?

Thinks

Reproduction

None

Logs

No response

System Info

None

Who can help?

@sayakpaul

[sayakpaul]

Good observation! Thank you for bringing this up!

Yeah, ideally, a reversal of the following would be needed:

diffusers/src/diffusers/pipelines/stable_diffusion_3/pipeline_stable_diffusion_3.py

Line 893 in 0f0b531

latents = (latents / self.vae.config.scaling_factor) + self.vae.config.shift_factor

Do you want to give it a try and open a PR, perhaps? Happy to help you through the process.

[bendanzzc]

Thanks, I'd like to try

[sayakpaul]

Lovely. Thanks so much.

[CodeExplode]

I've just implemented this in my own training code which is largely based on the diffusers example, and it does seem to noticeably help with image crispness in some with/without tests on the same training data (though non-deterministic choice of seed, image ordering, prompt shuffling, etc).

[sayakpaul]

Do you wanna show some comparisons?

[CodeExplode]

I only kept one image sorry. I tried training with the character Ahsoka as the toughest example in my dataset.

The left is training without handling shift, right is with handling shift, on approximately the same prompt (with some shuffling) at about the same number of steps. Without applying shift they all looked blurry like the left sample (after a few epochs), whereas with shift there was a mix of blurry and crisp previews, so it seemed to be helping. The samples were always generated with shift from the start as they use different code.

This was full finetuning, rather than LoRA training.

[sayakpaul]

Fixed #8917