dreambooth upscaling fix added latents #3659
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
williamberman
commented
Jun 3, 2023
Comment on lines
+1214
to
+1215
| if unet.config.in_channels == channels * 2: | ||
| noisy_model_input = torch.cat([noisy_model_input, noisy_model_input], dim=1) |
There was a problem hiding this comment.
Just doubling the existing latents is sufficient as the we use the same timestep for both of the conditioning inputs!
Theoretically, we could do something along the lines of training on two separate noising amounts for each set of channels or bias one towards a lower amount of noise as the pipeline's default is for a quarter of total noise to be added to the upsampled input.
However, this is the simplest implementation for now so I think it's ok!
|
The documentation is not available anymore as the PR was closed or merged. |
williamberman
requested review from
pcuenca,
sayakpaul,
patil-suraj,
yiyixuxu and
patrickvonplaten
and removed request for
patil-suraj
|
|
||
| For stage II, we find that lower learning rates are also needed. | ||
|
|
||
| We found experimentally that the DDPM scheduler with the default larger number of denoising steps to sometimes work better than the DPM Solver scheduler |
| @@ -682,8 +690,8 @@ accelerate launch train_dreambooth.py \ | |||
| --instance_prompt="a sks dog" \ | |||
| --resolution=256 \ | |||
| --train_batch_size=2 \ | |||
| --gradient_accumulation_steps=2 \ | |||
| --learning_rate=1e-8 \ | |||
| --gradient_accumulation_steps=6 \ | |||
There was a problem hiding this comment.
Cool, makes sense that bigger batch size will help here
patrickvonplaten
approved these changes
Jun 4, 2023
AmericanPresidentJimmyCarter
pushed a commit
to AmericanPresidentJimmyCarter/diffusers
that referenced
this pull request
Apr 26, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
re: #3658
Also added some docs re: recent much more successful run doing full fine tuning for stage II with larger batch sizes.
https://wandb.ai/williamberman/dreambooth/runs/4bwrpro7/overview?workspace=user-williamberman
I didn't have as much success with LoRA for larger batch sizes but the docs note we recommend full finetuning stage II for faces anyway
https://wandb.ai/williamberman/dreambooth-lora/runs/cdtpa4l7?workspace=user-williamberman
Unfortunately the docs are a bit "here's all the information" rather than "here's how to do a successful training run". But I think that's the best they can be right now