Use CC12M for LCM WDS training example #5908
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
Makes sense. I'd rather tackle that in a separate PR, if possible, so this one can be merged and helps the community with the issues linked in the first comment. |
patrickvonplaten
left a comment
patrickvonplaten
left a comment
There was a problem hiding this comment.
Great idea! @patil-suraj @dg845 can you please check here?
| for step, batch in enumerate(train_dataloader): | ||
| with accelerator.accumulate(unet): | ||
| image, text, _, _ = batch | ||
| image, text = batch |
There was a problem hiding this comment.
Just to confirm, is this change because the SD distillation script currently has a bug where it assumes that the Text2ImageDataset.train_dataloader batch consists of (image, text, orig_size, crop_coords) like the SDXL dataloader instead of just (image, text):
diffusers/examples/consistency_distillation/train_lcm_distill_sd_wds.py
Lines 176 to 178 in 110ac7f
and is otherwise unrelated to CC12M support?
There was a problem hiding this comment.
Looks like 711e468 mentions this. Feel free to close :).
There was a problem hiding this comment.
Yes, exactly, it's actually a bug and not directly related to CC12M :)
| --output_dir=$OUTPUT_DIR \ | ||
| --mixed_precision=fp16 \ | ||
| --resolution=512 \ | ||
| --learning_rate=1e-6 --loss_type="huber" --ema_decay=0.95 --adam_weight_decay=0.0 \ |
There was a problem hiding this comment.
Do the hyperparameters in the examples work well with the CC12M dataset or does it potentially make sense to revisit them?
There was a problem hiding this comment.
Not sure to be honest. I haven't experimented much with it, considering that CC12M is very small for current standards.
Perhaps it would be easier to include a disclaimer stating that this dataset is used for illustrative purposes and users are encouraged to bring their own.
|
I think it could be valuable to make sure |
I think that's a good idea! I'd rather work on that separately, if possible, so we can close those issues from the community. I'll add a disclaimer about the illustrative nature of the dataset. Edit: done, @dg845 let me know if that'd be enough. |
|
Thanks for the great reviews! |
* Fix SD scripts - there are only 2 items per batch * Adjustments to make the SDXL scripts work with other datasets * Use public webdataset dataset for examples * make style * Minor tweaks to the readmes. * Stress that the database is illustrative.
* Fix SD scripts - there are only 2 items per batch * Adjustments to make the SDXL scripts work with other datasets * Use public webdataset dataset for examples * make style * Minor tweaks to the readmes. * Stress that the database is illustrative.
Had to make some adjustments to make it work with CC12M, as the metadata field names in the json are different. In particular, if no
pwatermarkfield exists all images were rejected.cc @stevhliu @sayakpaul @patil-suraj
Fixes #5868, #5770.
Possibly fixes #5743 for other datasets.