Skip to content

[Stable Diffusion] Allow users to disable Safety checker if loading model from checkpoint #2768

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Mar 28, 2023

Conversation

Stax124
Copy link
Contributor

@Stax124 Stax124 commented Mar 21, 2023

  1. Allows users to cast down the pipeline to Float16, even if the checkpoint was saved in Float32 format.
  2. Allows users to disable loading the Safety checker and Feature extractor if users opts-out.

Benefits: The user will have the option to save both VRAM (safety checker and float16) and disk space (safety checker download)

Intended use:

from diffusers.pipelines.stable_diffusion.convert_from_ckpt import (
    download_from_original_stable_diffusion_ckpt,
)

pipe = download_from_original_stable_diffusion_ckpt(
    checkpoint_path="PATH_TO_CKPT",
    from_safetensors=use_safetensors,
    load_safety_checker=False,
    dtype=torch.float16,
)

pipe.to("cuda")
result = pipe("a dog")

Closes #2755

@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Mar 21, 2023

The documentation is not available anymore as the PR was closed or merged.

Copy link
Member

@sayakpaul sayakpaul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well done!

@sayakpaul
Copy link
Member

@patrickvonplaten could you check here?

@n00mkrad
Copy link

I'm surprised this was not a thing until now.

Does that mean saving a model as fp16 was not possible until now? Hope this gets merged asap.

@patrickvonplaten
Copy link
Contributor

I think this is a duplicate of: #2733 no?

@Stax124
Copy link
Contributor Author

Stax124 commented Mar 23, 2023

@patrickvonplaten I will try to check when I get home, thanks for pointing that out

@Stax124 Stax124 changed the title [Stable Diffusion] Enable casting to Float16 and disabling Safety checker if loading model from checkpoint [Stable Diffusion] Allow users to disable Safety checker if loading model from checkpoint Mar 23, 2023
@Stax124
Copy link
Contributor Author

Stax124 commented Mar 23, 2023

I had a look at the stuff you sent and you were correct, thanks for pointing that out - I wasn't aware that this was possible.
With that, I still believe that users should be able to at least disable the safety checker when loading the model - so I purged all the code for dtypes but I left the safety checker code in.

@Stax124
Copy link
Contributor Author

Stax124 commented Mar 25, 2023

Failed test is not caused by this PR as #2817 is facing the same issue

@patrickvonplaten
Copy link
Contributor

Thanks for adapting the PR - think we can merge this one

@patrickvonplaten patrickvonplaten merged commit 585f621 into huggingface:main Mar 28, 2023
w4ffl35 pushed a commit to w4ffl35/diffusers that referenced this pull request Apr 14, 2023
…odel from checkpoint (huggingface#2768)

* Allow user to disable SafetyChecker and enable dtypes if loading models from .ckpt or .safetensors

* Fix Import sorting (Ruff error)

* Get rid of the dtype convert method as it was implemented all along

* Fix the docstring

* Fix ruff formatting

---------

Co-authored-by: Patrick von Platen <[email protected]>
yoonseokjin pushed a commit to yoonseokjin/diffusers that referenced this pull request Dec 25, 2023
…odel from checkpoint (huggingface#2768)

* Allow user to disable SafetyChecker and enable dtypes if loading models from .ckpt or .safetensors

* Fix Import sorting (Ruff error)

* Get rid of the dtype convert method as it was implemented all along

* Fix the docstring

* Fix ruff formatting

---------

Co-authored-by: Patrick von Platen <[email protected]>
AmericanPresidentJimmyCarter pushed a commit to AmericanPresidentJimmyCarter/diffusers that referenced this pull request Apr 26, 2024
…odel from checkpoint (huggingface#2768)

* Allow user to disable SafetyChecker and enable dtypes if loading models from .ckpt or .safetensors

* Fix Import sorting (Ruff error)

* Get rid of the dtype convert method as it was implemented all along

* Fix the docstring

* Fix ruff formatting

---------

Co-authored-by: Patrick von Platen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug][StableDiffusion] Allow loading models with Float16 in load_pipeline_from_original_stable_diffusion_ckpt
5 participants