[Community Pipelines] lpw_stable_diffusion.py incompatible with Diffusers v0.10.0

[WASasquatch]

Describe the bug

When trying to load Stable Diffusion v2.0 with Diffusers v0.10.0 you are greeted with error below

ValueError: Pipeline <class 'diffusers_modules.git.lpw_stable_diffusion.StableDiffusionLongPromptWeightingPipeline'> expected {'tokenizer', 'text_encoder', 'unet', 'vae', 'safety_checker', 'feature_extractor', 'scheduler'}, but only {'vae', 'tokenizer', 'safety_checker', 'text_encoder', 'scheduler', 'unet'} were passed.

Reproduction

1. Load custom pipeline lpw_stable_diffusion in diffusers v0.10.0

Logs

D:\ProgramData\Anaconda3\lib\site-packages\diffusers\pipeline_utils.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    672         elif len(missing_modules) > 0:
    673             passed_modules = set(list(init_kwargs.keys()) + list(passed_class_obj.keys())) - optional_kwargs
--> 674             raise ValueError(
    675                 f"Pipeline {pipeline_class} expected {expected_modules}, but only {passed_modules} were passed."
    676             )

ValueError: Pipeline <class 'diffusers_modules.git.lpw_stable_diffusion.StableDiffusionLongPromptWeightingPipeline'> expected {'tokenizer', 'text_encoder', 'unet', 'vae', 'safety_checker', 'feature_extractor', 'scheduler'}, but only {'vae', 'tokenizer', 'safety_checker', 'text_encoder', 'scheduler', 'unet'} were passed.

System Info

• diffusers version: 0.10.0.dev0
• Platform: Windows-10-10.0.22621-SP0
• Python version: 3.9.13
• PyTorch version (GPU?): 1.12.1 (True)
• Huggingface_hub version: 0.11.0
• Transformers version: 4.24.0
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

[WASasquatch]

I seem to have inadvertently fixed whatever the issue was with adding a couple fixed for Stable Diffusion v2.0

[WASasquatch]

Actually, I take that back. The issue still persists. It just only gives the error when using Stable Diffusion v2.0 model it seems for some reason.

latest logs

~\AppData\Local\Temp\ipykernel_29864\1302006958.py in <module>
      9 
     10     # Instanitate Idun Class
---> 11     idun = Idun(
     12 
     13         diffusers_repo = 'bleeding',

~\AppData\Local\Temp\ipykernel_29864\1356244608.py in __init__(self, **opts)
     78 
     79             # Initialize Pipe
---> 80             self.createPipe(**opts)
     81 
     82             # Set Samplers

~\AppData\Local\Temp\ipykernel_29864\1356244608.py in createPipe(self, **opts)
    334 
    335         # Get a pipe
--> 336         self.pipe = self.getPipe(self.model_name, model_type)
    337 
    338         # Use device map

~\AppData\Local\Temp\ipykernel_29864\1356244608.py in getPipe(self, model, model_type)
    282             model_id = self.model_ids[model]
    283 
--> 284             self.pipe = DiffusionPipeline.from_pretrained(model_id, **conf)
    285 
    286         return self.pipe

~\AppData\Roaming\Python\Python39\site-packages\diffusers\pipeline_utils.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    699         elif len(missing_modules) > 0:
    700             passed_modules = set(list(init_kwargs.keys()) + list(passed_class_obj.keys())) - optional_kwargs
--> 701             raise ValueError(
    702                 f"Pipeline {pipeline_class} expected {expected_modules}, but only {passed_modules} were passed."
    703             )

ValueError: Pipeline <class 'diffusers_modules.git.lpw_stable_diffusion.StableDiffusionLongPromptWeightingPipeline'> expected {'scheduler', 'feature_extractor', 'safety_checker', 'text_encoder', 'unet', 'tokenizer', 'vae'}, but only {'unet', 'scheduler', 'safety_checker', 'tokenizer', 'vae', 'text_encoder'} were passed.

Configuration Applies (Full conf)

        self.model_configs = opts.pop('model_configs', {
            'full': {
                    'model_id': next(iter(self.model_ids)),
                    'custom_pipeline': opts.get('custom_pipeline', 'lpw_stable_diffusion'),
                    'safety_checker': None,
                    'use_auth_token': self.token,
                    'use_device_map': 'auto',
                    #'local_files_only': True
            },
            'half': {
                    'model_id': next(iter(self.model_ids)),
                    'custom_pipeline': opts.get('custom_pipeline', 'lpw_stable_diffusion'),
                    'safety_checker': None,
                    'use_auth_token': self.token,
                    'use_device_map': 'auto',
                    'revision': 'fp16',
                    'torch_dtype': torch.float16,
                    #'local_files_only': True
            }
        })

[adhikjoshi]

Have the same issue, it turns out lpw_stable_diffusion is not compatible with latest version of diffusers nor with new version of transformers. Its breaking changes.

#1536

[WASasquatch]

Have the same issue, it turns out lpw_stable_diffusion is not compatible with latest version of diffusers nor with new version of transformers. Its breaking changes.

#1536

I feel this just adds to the whole thing that Community Pipelines is just simply a bad approach. They're adopted into diffusers, but are not maintained by diffusers. They'll all quickly become broken pipes at the rate of development and require community action, if anyone bothers. Authors won't be getting issue notifications regarding possible issues, as it's on diffusers.

However, on v0.10.0 I am able to use LPW pipe fine, with redshift diffusion, and v1.5 of SD. When I load v2.0 of SD, that's when I encounter an error.

[Skquark]

A workaround fix for that is to add feature_extractor=None to your pipeline calls. Also if you're using enable_attention_slicing, the lpw pipe is not updated with the fix for slice_size when using SD2. I have my own custom lpw pipeline with that updated on huggingface which you can load with custom_pipeline="AlanB/lpw_stable_diffusion_mod" if you wanna borrow it until the official correction is make.
I'm still having issues with 3 of the schedulers generating those ugly mosaic images using lpw with v2.0 and I don't know the cause/fix for that problem yet... Just patiently waiting while the dust settles.

[WASasquatch]

After having tested 2.0 pretty extensively now, I'm jut not too interested in it. Terribly botched in so many aspects. So that isn't much of an issue now, though I am sure many people are going to use LPW with 2.0, as it's one of the best pipes for composing. Haven't even used official diffusers pipes since LPW was released. No reason to since weighting is absolutely crucial to image synthesis.

[patrickvonplaten]

Gently pinging @SkyTNT here

[anton-l]

Fixed by @SkyTNT in #1570