-
Notifications
You must be signed in to change notification settings - Fork 6.6k
Description
Describe the bug
Can not run LCM distillation example script due to aws dataset access
Reproduction
runwayml/stable-diffusion-v1-5
PROGRAM="train_lcm_distill_lora_sd_wds.py
--pretrained_teacher_model=$MODEL_DIR
--output_dir=$OUTPUT_DIR
--mixed_precision=fp16
--resolution=512
--lora_rank=64
--learning_rate=1e-6 --loss_type="huber" --adam_weight_decay=0.0
--max_train_steps=1000
--max_train_samples=4000000
--dataloader_num_workers=8
--train_shards_path_or_url='pipe:aws s3 cp s3://muse-datasets/laion-aesthetic6plus-min512-data/{00000..01210}.tar -'
--validation_steps=200
--checkpointing_steps=200 --checkpoints_total_limit=10
--train_batch_size=12
--gradient_checkpointing --enable_xformers_memory_efficient_attention
--gradient_accumulation_steps=1
--use_8bit_adam
--resume_from_checkpoint=latest
--report_to=wandb
--seed=453645634
--push_to_hub \
Logs
miniconda3/lib/python3.11/site-packages/webdataset/handlers.py:33: UserWarning: OSError("
(('aws s3 cp s3://muse-datasets/laion-aesthetic6plus-min512-data/00627.tar -',), {'shell': True, 'bufsi
ze': 8192}): exit 1 (read) {}", <webdataset.gopen.Pipe object at 0x7fa585ffd6d0>, 'pipe:aws s3 cp s3://
muse-datasets/laion-aesthetic6plus-min512-data/00627.tar -')
warnings.warn(repr(exn))System Info
Ubuntu
Who can help?
@patil-suraj I saw your commit in the script, wonder if you have any idea about the access issue.
Or is there a quick workaround to unblock me from running the example? Thank you!