SD3 wrong VAE encode in training script

[JinxuXiang]

diffusers/examples/dreambooth/train_dreambooth_lora_sd3.py

Line 1639 in c1dc2ae

model_input = model_input * vae.config.scaling_factor

Should it be:

model_input = (model_input - vae.config.shift_factor) * vae.config.scaling_factor

[wangqixun]

Great attention to detail! The code we trained for the sd3 controlnet was written based on this, so it should also have this bug.
@haofanwang

[bghira]

i am guessing that needs to be applied during decode in the pipelines as well? (it is)

            latents = (
                latents / self.vae.config.scaling_factor
            ) + self.vae.config.shift_factor

[sayakpaul]

@JinxuXiang Yeah you're right. Would you mind submitting a PR? Thank you very much.