[2737]: Add DPMSolverMultistepScheduler to CLIP guided community pipeline

[nipunjindal]

Enabled DPMSolverMultistepScheduler in CLIP-guided pipeline.
Issue: #2737

Here is code to test the changes:

import torch
from diffusers import (
    DiffusionPipeline,
    DPMSolverMultistepScheduler,
    LMSDiscreteScheduler,
)
from transformers import CLIPFeatureExtractor, CLIPModel

feature_extractor = CLIPFeatureExtractor.from_pretrained(
    "laion/CLIP-ViT-B-32-laion2B-s34B-b79K"
)
clip_model = CLIPModel.from_pretrained(
    "laion/CLIP-ViT-B-32-laion2B-s34B-b79K", torch_dtype=torch.float16
)


guided_pipeline = DiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    custom_pipeline="/home/njindal/diffusers/examples/community/clip_guided_stable_diffusion.py",
    clip_model=clip_model,
    feature_extractor=feature_extractor,
    torch_dtype=torch.float16,
)
guided_pipeline.scheduler = DPMSolverMultistepScheduler.from_config(
    guided_pipeline.scheduler.config
)

guided_pipeline.enable_attention_slicing()
guided_pipeline = guided_pipeline.to("cuda")

prompt = "fantasy book cover, full moon, fantasy forest landscape, golden vector elements, fantasy magic, dark light night, intricate, elegant, sharp focus, illustration, highly detailed, digital painting, concept art, matte, art by WLOP and Artgerm and Albert Bierstadt, masterpiece"

generator = torch.Generator(device="cuda").manual_seed(0)

image = guided_pipeline(
    prompt=prompt,
    num_inference_steps=10,
    guidance_scale=7.5,
    clip_guidance_scale=100,
    num_cutouts=4,
    use_cutouts=False,
    generator=generator,
).images[0]

display(image)

Within 10 steps able to get good result with new scheduler

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[yuvalkirstain]

@nipunjindal thank you! so it seems like the process for DPMSolverMultistepScheduler is the same as PNDMScheduler.

Do you happen to know what those lines are meant to do?

fac = torch.sqrt(beta_prod_t)
sample = pred_original_sample * (fac) + latents * (1 - fac)

and why does it use the cutouts + spherical loss rather than the loss from the original paper?

Also, it probably worth adding a few side-by-side image of w/o classifier guidance vs w classifier guidance (for different classifier guidance scales).

[pete-finesse]

What would be the process for including the ancestral samplers as well? I find these work best when using clip guidance using the sdk from stability.ai

[sayakpaul]

Do you mind elaborating on this change?

[patrickvonplaten]

This looks correct! This should not be done in the pipeline :-)

[sayakpaul]

LGTM, well done 🔥

Could you maybe also include a few lines about this support in the README as well since it improves the results (efficiency-wise at least). Do you think that would make sense?

@patrickvonplaten could you give it a quick review too?

[sayakpaul]

What would be the process for including the ancestral samplers as well? I find these work best when using clip guidance using the sdk from stability.ai

The process would not differ much. You will need to consider adding an ancestral sampler in the condition (such as this). If it's not immediately compatible, then some changes might be necessary. An example:

diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_instruct_pix2pix.py

Line 344 in 92e1164

# For karras style schedulers the model does classifer free guidance using the

[patrickvonplaten]

Ok to merge for me