[Variant] Add "variant" as input kwarg so to have better UX when downloading no_ema or fp16 weights #2305

patrickvonplaten · 2023-02-09T13:59:23Z

This PR adds a "variant" keyword argument so that model variations can be better stored on the "main" branch.

Important: See discussion here: #1764 also

It's the mirror of huggingface/transformers#21332 for diffusers.

Make sure you're using transformers on "main" when trying out the following:

from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained("hf-internal-testing/stable-diffusion-all-variants")
pipe = DiffusionPipeline.from_pretrained("hf-internal-testing/stable-diffusion-all-variants", variant="fp16")
pipe = DiffusionPipeline.from_pretrained("hf-internal-testing/stable-diffusion-all-variants", variant="no_ema")

These commands will load the respective variants from: https://huggingface.co/hf-internal-testing/stable-diffusion-all-variants/tree/main

Important:

It is made sure that variant loading works locally
It is made sure that loading works when internet is done and local_files_only=True
It is made sure that only non-variant files are downloaded if both non-variant files and variant files are present
It is made sure that only variant files are downloaded if both non-variant files and variant files are present
It is made sure that a mixture of variant and non-variant files can be loaded if not all files are present as variations (e.g. "no_ema")

To achieve this the ignore and allow patterns logic is refactored and made simpler, more precise.

Please have a look at the tests for more details.

Deprecation Cycle

Note that while we should merge this PR we probably need to wait 1,2 releases until we add the model variants to popular repos such as stable-diffusion-v1-4 , v1-5, v2-0 and v2-1 as diffusers < 0.13.0dev0 would otherwise downloads GB of unused models. That's why there are some "in-the-future" deprecation warnings in the PR here.

Stats about model repos having variants.

We only have ~55 public models (from 3800 models) that use branches as variations, so I think we could relatively easily transition all of those in ~1 month. Here the list:

google/ncsnpp-ffhq-1024: ['master']
google/ncsnpp-ffhq-1024: ['master']
shalpin87/diffusion_conditional: ['backup']
CompVis/stable-diffusion-v1-3: ['fp16']
CompVis/stable-diffusion-v1-1: ['fp16']
CompVis/stable-diffusion-v1-2: ['fp16']
CompVis/stable-diffusion-v1-4: ['non-ema', 'onnx', 'bf16', 'flax', 'fp16']
hakurei/waifu-diffusion: ['fp16']
rinna/japanese-stable-diffusion: ['fp16']
naclbit/trinart_stable_diffusion_v2: ['diffusers-95k', 'diffusers-60k', 'diffusers-115k']
pcuenq/stable-diffusion-v1-4: ['onnx']
lambdalabs/sd-pokemon-diffusers: ['onnx']
CompVis/stable-diffusion-v1-5: ['fp16']
Gazoche/sd-gundam-diffusers: ['epoch-000020', 'epoch-000081', 'epoch-000025']
runwayml/stable-diffusion-inpainting: ['onnx', 'fp16']
fusing/sd-inpaint-temp: ['fp16']
runwayml/stable-diffusion-v1-5: ['onnx', 'fp16', 'non-ema', 'flax', 'bf16']
ckpt/sd15: ['flax', 'bf16', 'fp16']
aarondotwork/sd-pokemon-diffusers: ['fp16']
technillogue/waifu-diffusion: ['fp16']
DGSpitzer/Cyberpunk-Anime-Diffusion: ['fp16']
uripper/GIANNIS: ['ONNX', 'Traced', 'ONNX-Q']
microsoft/vq-diffusion-ithq: ['fp16']
fusing/rdm: ['fp16']
CompVis/ldm-super-resolution-4x-openimages: ['fp16']
lilpotat/f2: ['flax']
lilpotat/a3: ['flax']
lilpotat/rbm: ['flax']
BAAI/AltDiffusion: ['fp16']
fusing/test: ['fp16']
stabilityai/stable-diffusion-2: ['fp16', 'bf16']
stabilityai/stable-diffusion-2-base: ['onnx', 'fp16']
stabilityai/stable-diffusion-2-depth: ['fp16']
stabilityai/stable-diffusion-2-inpainting: ['fp16']
stabilityai/stable-diffusion-x4-upscaler: ['fp16']
Abhilashvj/openjourney_copy: ['master']
questcoast/clone-wars-diffusion-v1: ['readme']
lilpotat/ashleymoore: ['flax']
jplumail/matthieu-v1-pipe: ['fp16']
stabilityai/stable-diffusion-2-1: ['bf16', 'fp16']
stabilityai/stable-diffusion-2-1-base: ['fp16']
jplumail/matthieu-v2-pipe: ['fp16']
NickKolok/arikioyami-20221223_1: ['backforward_overfit']
mann-e/mann-e: ['master']
DucHaiten/DucHaitenAIart: ['safetensors']
ShibaDeveloper/olivia-v1.0: ['safetensors']
cdefghijkl/luber: ['safetensors']
mddy/abyss2-diffusers: ['master']
OFA-Sys/small-stable-diffusion-v0: ['onnx']
timbrooks/instruct-pix2pix: ['fp16']
DucHaiten/DucHaitenAnime: ['vae']
neemspees/dnd-maps-2: ['v2', 'v3']
ruiruin/counmargemodel: ['fp16']
Nacholmo/AbyssOrangeMix2-hard-vae-swapped: ['fp16']
Nacholmo/Counterfeit-V2.5-vae-swapped: ['de_vae', 'dvae', 'fp16']
Nacholmo/VOXO-v0-vtuber-diffusers: ['fp16']

Final TODOs

Add nice docs
Deprecate "revision="fp16" in the smoothest way possible
Make sure tests are not becoming too slow

HuggingFaceDocBuilderDev · 2023-02-09T14:03:44Z

The documentation is not available anymore as the PR was closed or merged.

src/diffusers/models/modeling_utils.py

src/diffusers/pipelines/pipeline_utils.py

src/diffusers/models/modeling_utils.py

williamberman · 2023-02-10T22:18:19Z

src/diffusers/models/modeling_utils.py

@@ -89,12 +89,12 @@ def find_tensor_attributes(module: torch.nn.Module) -> List[Tuple[str, Tensor]]:
        return first_tuple[1].dtype


-def load_state_dict(checkpoint_file: Union[str, os.PathLike]):
+def load_state_dict(checkpoint_file: Union[str, os.PathLike], variant: Optional[str] = None):


we could directly check the extension of checkpoint_file to know whether or not to load with safetensors. That way we wouldn't have to pass in variant

src/diffusers/pipelines/pipeline_utils.py

…into add_variant

pcuenca

Some doc nits and typos.

docs/source/en/using-diffusers/loading.mdx

pcuenca · 2023-02-14T13:14:44Z

docs/source/en/using-diffusers/loading.mdx

+```
+
+Now all model components of the pipeline are stored in half-precision dtype. We can now save the 
+pipeline under a `"fp16"` variant as follows:


Not sure if we should standardize / recommend fp16 (that's the name of the old revision branch), or float16 (the torch type).

Good question! Think it'll be difficult to nudge the community towards float16 given that we advertised ="fp16" everywhere - also in terms of deprecating it'll be difficult

@williamberman @patil-suraj wdyt?

Agree with Patrick here, I think using fp16 is pretty standard now. And I think it's okay to have the same name for revision and variant here as essentially variant is a better alternative to revision.

+1 I think we've stadardized on fp16

I think if we also want to have sets of equivalent variations that's cool too.

i.e. if the user passes "float16" as the variant, we use fp16 anyway and log letting them know

docs/source/en/using-diffusers/loading.mdx

williamberman · 2023-02-15T19:07:58Z

docs/source/en/using-diffusers/loading.mdx

+
+throws an Exception:
+```
+OSError: Error no file named diffusion_pytorch_model.bin found in directory ./stable-diffusion-v1-45/vae since we **only** stored the model 


We should throw a better error here, something like

"Error: You loaded the pipeline without a variant. Only found variants: fp16, bf16 in the repository."

And similarly, when loading with a variant that isn't present.

"Error: You loaded the pipeline with variant: fp16. Only found variants: bf16, no_ema in the repository."

We could do this as a follow up PR?

docs/source/en/using-diffusers/loading.mdx

pcuenca · 2023-02-15T18:51:20Z

src/diffusers/pipelines/pipeline_utils.py

+                    onnx_variant_filenames = set([f for f in variant_filenames if f.endswith(".onnx")])
+                    onnx_model_filenames = set([f for f in model_filenames if f.endswith(".onnx")])
+                    if len(onnx_variant_filenames) > 0 and onnx_model_filenames != onnx_variant_filenames:
+                        logger.warn(
+                            f"\nA mixture of {variant} and non-{variant} filenames will be loaded.\nLoaded {variant} filenames:\n[{', '.join(onnx_variant_filenames)}]\nLoaded non-{variant} filenames:\n[{', '.join(onnx_model_filenames - onnx_variant_filenames)}\nIf this behavior is not expected, please check your folder structure."
+                        )


I still think this should deal with the .safetensors extension, not onnx.

src/diffusers/pipelines/pipeline_utils.py

src/diffusers/models/modeling_utils.py

williamberman · 2023-02-15T19:26:01Z

src/diffusers/models/modeling_utils.py

+                        f"You are loading the variant {variant} from {pretrained_model_name_or_path} via `revision='{variant}'` even though you can load it via `variant=`{variant}`. Loading model variants via `revision='{variant}'` is deprecated and will be removed in diffusers v1. Please use `variant='{variant}'` instead. For more information, please have a look at: ",
+                        FutureWarning,
+                    )
+                except:  # noqa: E722


Shouldn't we only catch EntryNotFoundError in this case?

Think pure except is a bit safer here, just to be sure we don't miss a potential other type of error. Code will be removed somewhat soonish anyways though as it's just to maintain (future) deprecated behavior.

williamberman · 2023-02-15T19:27:07Z

src/diffusers/models/modeling_utils.py

+                        revision=revision,
+                    )
+                    warnings.warn(
+                        f"You are loading the variant {variant} from {pretrained_model_name_or_path} via `revision='{variant}'` even though you can load it via `variant=`{variant}`. Loading model variants via `revision='{variant}'` is deprecated and will be removed in diffusers v1. Please use `variant='{variant}'` instead. For more information, please have a look at: ",


Is this log message incomplete? "please have a look at: " < I think something goes at the end here?

williamberman · 2023-02-15T19:32:43Z

src/diffusers/models/modeling_utils.py

+            if revision in DEPRECATED_REVISION_ARGS and version.parse(
+                version.parse(__version__).base_version
+            ) >= version.parse("0.15.0"):
+                variant = _add_variant(weights_name, revision)
+
+                try:
+                    model_file = hf_hub_download(
+                        pretrained_model_name_or_path,
+                        filename=weights_name,
+                        cache_dir=cache_dir,
+                        force_download=force_download,
+                        proxies=proxies,
+                        resume_download=resume_download,
+                        local_files_only=local_files_only,
+                        use_auth_token=use_auth_token,
+                        user_agent=user_agent,
+                        subfolder=subfolder,
+                        revision=revision,
+                    )
+                    warnings.warn(
+                        f"You are loading the variant {variant} from {pretrained_model_name_or_path} via `revision='{variant}'` even though you can load it via `variant=`{variant}`. Loading model variants via `revision='{variant}'` is deprecated and will be removed in diffusers v1. Please use `variant='{variant}'` instead. For more information, please have a look at: ",
+                        FutureWarning,
+                    )
+                except:  # noqa: E722
+                    warnings.warn(
+                        f"You are loading the variant {variant} from {pretrained_model_name_or_path} via `revision='{variant}'`. This behavior is deprecated and will be removed in diffusers v1. One should use `variant='{variant}'` instead. However, it appears that {pretrained_model_name_or_path} currently does not have a {_add_variant(weights_name)} file in the 'main' branch of {pretrained_model_name_or_path}. \n The Diffusers team and community would be very grateful if you could open an issue: https://github.com/huggingface/diffusers/issues/new with the title '{pretrained_model_name_or_path} is missing {_add_variant(weights_name)}' so that the correct variant file can be added.",
+                        FutureWarning,
+                    )
+                    model_file = None
+            else:
+                # Load from URL or cache if already cached
+                model_file = hf_hub_download(
+                    pretrained_model_name_or_path,
+                    filename=weights_name,
+                    cache_dir=cache_dir,
+                    force_download=force_download,
+                    proxies=proxies,
+                    resume_download=resume_download,
+                    local_files_only=local_files_only,
+                    use_auth_token=use_auth_token,
+                    user_agent=user_agent,
+                    subfolder=subfolder,
+                    revision=revision,
+                )
            return model_file


I'm a bit confused by the logic here. I think the second error message about "main" not having a file seems to indicate we're checking the main branch but we still use the passed in revision to check a branch? Maybe I'm missing something

Yes, so essentially we should deprecate the behavior of doing revision="fp16". Now we might have cases where this works because the branch still exists, but we do want to tell the user that this behavior is deprecated, so we throw a warning.

Now there are two possibilities:

1.) There is already a "fp16" variant file on the main branch -> in this case the user can be directly guided to the new usage

2.) There is not yet a "fp16" variant file. In this case we probably should do it manually, therefore I'm trying to have the user open an issue

You're 100% right @williamberman - the code was bad here. Refactored it, should be better now I hope

src/diffusers/models/modeling_utils.py

docs/source/en/using-diffusers/loading.mdx

Co-authored-by: Suraj Patil <[email protected]>

docs/source/en/using-diffusers/loading.mdx

Co-authored-by: Suraj Patil <[email protected]>

Co-authored-by: Will Berman <[email protected]> Co-authored-by: Suraj Patil <[email protected]>

…loading no_ema or fp16 weights (huggingface#2305) * [Variant] Add variant loading mechanism * clean * improve further * up * add tests * add some first tests * up * up * use path splittetx * add deprecate * deprecation warnings * improve docs * up * up * up * fix tests * Apply suggestions from code review Co-authored-by: Pedro Cuenca <[email protected]> * Apply suggestions from code review Co-authored-by: Pedro Cuenca <[email protected]> * correct code format * fix warning * finish * Apply suggestions from code review Co-authored-by: Suraj Patil <[email protected]> * Apply suggestions from code review Co-authored-by: Suraj Patil <[email protected]> * Update docs/source/en/using-diffusers/loading.mdx Co-authored-by: Suraj Patil <[email protected]> * Apply suggestions from code review Co-authored-by: Will Berman <[email protected]> Co-authored-by: Suraj Patil <[email protected]> * correct loading docs * finish --------- Co-authored-by: Pedro Cuenca <[email protected]> Co-authored-by: Suraj Patil <[email protected]> Co-authored-by: Will Berman <[email protected]>

patrickvonplaten added 2 commits February 9, 2023 15:58

[Variant] Add variant loading mechanism

ab9ef6a

clean

91ee04e

patrickvonplaten added 6 commits February 9, 2023 18:02

improve further

0b45377

up

cbe2066

add tests

c760708

add some first tests

8d77537

up

4f6d13c

up

e329951

patrickvonplaten changed the title ~~Add variant~~ [Variant] Add "variant" as input kwarg so to have better UX when downloading no_ema or fp16 weights Feb 10, 2023

Merge branch 'main' into add_variant

847bc0f

patrickvonplaten requested review from williamberman, pcuenca, patil-suraj and yiyixuxu February 10, 2023 15:21

pcuenca reviewed Feb 10, 2023

View reviewed changes

src/diffusers/models/modeling_utils.py Show resolved Hide resolved

pcuenca reviewed Feb 10, 2023

View reviewed changes

src/diffusers/pipelines/pipeline_utils.py Outdated Show resolved Hide resolved

williamberman reviewed Feb 10, 2023

View reviewed changes

src/diffusers/models/modeling_utils.py Show resolved Hide resolved

williamberman reviewed Feb 10, 2023

View reviewed changes

williamberman reviewed Feb 11, 2023

View reviewed changes

src/diffusers/pipelines/pipeline_utils.py Show resolved Hide resolved

patrickvonplaten mentioned this pull request Feb 13, 2023

is_safetensors_compatible refactor #2316

Closed

patrickvonplaten added 8 commits February 13, 2023 17:11

use path splittetx

710480d

Merge branch 'main' into add_variant

b506882

add deprecate

9262bbf

deprecation warnings

4a0ff60

Merge branch 'add_variant' of https://github.com/huggingface/diffusers …

04622d2

…into add_variant

improve docs

010f2ed

up

bdebb36

up

73bf79f

pcuenca reviewed Feb 14, 2023

View reviewed changes

williamberman reviewed Feb 15, 2023

View reviewed changes

docs/source/en/using-diffusers/loading.mdx Outdated Show resolved Hide resolved

williamberman reviewed Feb 15, 2023

View reviewed changes

docs/source/en/using-diffusers/loading.mdx Outdated Show resolved Hide resolved

williamberman reviewed Feb 15, 2023

View reviewed changes

docs/source/en/using-diffusers/loading.mdx Outdated Show resolved Hide resolved

williamberman reviewed Feb 15, 2023

View reviewed changes

docs/source/en/using-diffusers/loading.mdx Outdated Show resolved Hide resolved

pcuenca reviewed Feb 15, 2023

View reviewed changes

williamberman reviewed Feb 15, 2023

View reviewed changes

src/diffusers/models/modeling_utils.py Outdated Show resolved Hide resolved

patrickvonplaten commented Feb 16, 2023

View reviewed changes

docs/source/en/using-diffusers/loading.mdx Outdated Show resolved Hide resolved

Apply suggestions from code review

f26abeb

Co-authored-by: Suraj Patil <[email protected]>

patrickvonplaten commented Feb 16, 2023

View reviewed changes

docs/source/en/using-diffusers/loading.mdx Outdated Show resolved Hide resolved

patrickvonplaten and others added 6 commits February 16, 2023 10:01

Apply suggestions from code review

57fbe8f

Co-authored-by: Suraj Patil <[email protected]>

Update docs/source/en/using-diffusers/loading.mdx

cbffa77

Co-authored-by: Suraj Patil <[email protected]>

Apply suggestions from code review

fb22078

Co-authored-by: Will Berman <[email protected]> Co-authored-by: Suraj Patil <[email protected]>

correct loading docs

dbdd126

finish

5bcd411

Merge branch 'main' into add_variant

f7b9fb7

patrickvonplaten merged commit e5810e6 into main Feb 16, 2023

patrickvonplaten deleted the add_variant branch February 16, 2023 10:03

This was referenced Feb 19, 2023

Fix deprecation warning #2426

Merged

CompVis/stable-diffusion-v1-4 is missing fp16 files #2419

Closed

patrickvonplaten mentioned this pull request Feb 27, 2023

Adding use_safetensors argument to give more control to users #2123

Merged

pcuenca mentioned this pull request Mar 6, 2023

train_text_to_image_flax.py no flax_model.msgpack or pytorch_model.bin #2410

Closed

This was referenced Mar 9, 2023

stabilityai/stable-diffusion-2-depth is missing diffusion_py torch_model.bin #2611

Closed

Provide a way to find out which model branches map to which pipelines #2651

Closed

[Variant] Add "variant" as input kwarg so to have better UX when downloading no_ema or fp16 weights #2305

[Variant] Add "variant" as input kwarg so to have better UX when downloading no_ema or fp16 weights #2305

Uh oh!

Conversation

patrickvonplaten commented Feb 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deprecation Cycle

Stats about model repos having variants.

Uh oh!

HuggingFaceDocBuilderDev commented Feb 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pcuenca left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

patrickvonplaten commented Feb 9, 2023 •

edited

Loading

HuggingFaceDocBuilderDev commented Feb 9, 2023 •

edited

Loading