Adding `use_safetensors` argument to give more control to users #2123

Narsil · 2023-01-26T16:19:54Z

about which weights they use.

HuggingFaceDocBuilderDev · 2023-01-26T16:28:07Z

The documentation is not available anymore as the PR was closed or merged.

patrickvonplaten · 2023-01-26T18:45:02Z

I understand the need for the PR, but I wonder whether we could use the soon-to-be-added "variant" kwarg for this instead to not add too many kwargs.

The idea with "variant" is summarized here: #1764

I think it might be better to handle all of this with the "variant" kwarg instead. E.g. from_pretrained(variant="safetensors") then safetensors has to be installed and the files have to be there or an error is thrown.

We need to also align this with transformers cc @sgugger @LysandreJik .

sgugger · 2023-01-26T19:03:53Z

For Transformers, safetensors is the default if available, there is no special flag for it and we are not planning on adding one. We just look at model.safetensors before anything else.

patrickvonplaten · 2023-01-27T16:55:02Z

src/diffusers/models/modeling_utils.py

@@ -350,6 +350,11 @@ def from_pretrained(cls, pretrained_model_name_or_path: Optional[Union[str, os.P
                also tries to not use more than 1x model size in CPU memory (including peak memory) while loading the
                model. This is only supported when torch version >= 1.9.0. If you are using an older version of torch,
                setting this argument to `True` will raise an error.
+            use_safetensors (`bool`, *optional*, defaults to `None`):


Suggested change

use_safetensors (`bool`, *optional*, defaults to `None`):

use_safetensors (`bool`, *optional*):

patrickvonplaten

Actually upon section reflection, I think this makes sense.

Can we also add tests though?
And also would maybe save_loading suit better (since we have safe_serialization ?)

patrickvonplaten · 2023-01-27T16:57:40Z

src/diffusers/models/modeling_utils.py

@@ -350,6 +350,11 @@ def from_pretrained(cls, pretrained_model_name_or_path: Optional[Union[str, os.P
                also tries to not use more than 1x model size in CPU memory (including peak memory) while loading the
                model. This is only supported when torch version >= 1.9.0. If you are using an older version of torch,
                setting this argument to `True` will raise an error.
+            use_safetensors (`bool`, *optional*, defaults to `None`):
+                If set to `True`, the pipeline will forcibly load the models using `safetensors` weights. If set to


Suggested change

If set to `True`, the pipeline will forcibly load the models using `safetensors` weights. If set to

If set to `True`, the pipeline will forcibly load the model from `safetensors` weights. If set to

patrickvonplaten · 2023-01-27T16:57:55Z

src/diffusers/models/modeling_utils.py

@@ -350,6 +350,11 @@ def from_pretrained(cls, pretrained_model_name_or_path: Optional[Union[str, os.P
                also tries to not use more than 1x model size in CPU memory (including peak memory) while loading the
                model. This is only supported when torch version >= 1.9.0. If you are using an older version of torch,
                setting this argument to `True` will raise an error.
+            use_safetensors (`bool`, *optional*, defaults to `None`):
+                If set to `True`, the pipeline will forcibly load the models using `safetensors` weights. If set to
+                `None` (the default). The pipeline will load using `safetensors` if the safetensors weights are


Suggested change

`None` (the default). The pipeline will load using `safetensors` if the safetensors weights are

`None` (the default). The pipeline will load using `safetensors` if safetensors weights are

patrickvonplaten · 2023-01-27T16:58:22Z

src/diffusers/models/modeling_utils.py

+            use_safetensors (`bool`, *optional*, defaults to `None`):
+                If set to `True`, the pipeline will forcibly load the models using `safetensors` weights. If set to
+                `None` (the default). The pipeline will load using `safetensors` if the safetensors weights are
+                actually available *and* you have the library installed. If the to `False` the pipeline will *not* use


Suggested change

actually available *and* you have the library installed. If the to `False` the pipeline will *not* use

available *and* if `safetensors` is installed. If set to `False` the pipeline will *not* use

patrickvonplaten · 2023-01-27T16:58:34Z

src/diffusers/models/modeling_utils.py

+                If set to `True`, the pipeline will forcibly load the models using `safetensors` weights. If set to
+                `None` (the default). The pipeline will load using `safetensors` if the safetensors weights are
+                actually available *and* you have the library installed. If the to `False` the pipeline will *not* use
+                `safetensors` at all.


Suggested change

`safetensors` at all.

`safetensors`.

patrickvonplaten · 2023-01-27T16:58:43Z

src/diffusers/pipelines/pipeline_utils.py

@@ -400,6 +400,11 @@ def from_pretrained(cls, pretrained_model_name_or_path: Optional[Union[str, os.P
                setting this argument to `True` will raise an error.
            return_cached_folder (`bool`, *optional*, defaults to `False`):
                If set to `True`, path to downloaded cached folder will be returned in addition to loaded pipeline.
+            use_safetensors (`bool`, *optional*, defaults to `None`):


Suggested change

use_safetensors (`bool`, *optional*, defaults to `None`):

use_safetensors (`bool`, *optional*):

patrickvonplaten · 2023-01-27T16:59:16Z

src/diffusers/pipelines/pipeline_utils.py

@@ -400,6 +400,11 @@ def from_pretrained(cls, pretrained_model_name_or_path: Optional[Union[str, os.P
                setting this argument to `True` will raise an error.
            return_cached_folder (`bool`, *optional*, defaults to `False`):
                If set to `True`, path to downloaded cached folder will be returned in addition to loaded pipeline.
+            use_safetensors (`bool`, *optional*, defaults to `None`):
+                If set to `True`, the pipeline will forcibly load the models using `safetensors` weights. If set to
+                `None` (the default). The pipeline will load using `safetensors` if the safetensors weights are


Suggested change

`None` (the default). The pipeline will load using `safetensors` if the safetensors weights are

`None` (the default), the pipeline will be loaded from `safetensors` if the safetensors weights are

patrickvonplaten · 2023-01-27T16:59:33Z

src/diffusers/pipelines/pipeline_utils.py

+            use_safetensors (`bool`, *optional*, defaults to `None`):
+                If set to `True`, the pipeline will forcibly load the models using `safetensors` weights. If set to
+                `None` (the default). The pipeline will load using `safetensors` if the safetensors weights are
+                actually available *and* you have the library installed. If the to `False` the pipeline will *not* use


Suggested change

actually available *and* you have the library installed. If the to `False` the pipeline will *not* use

available *and* if `safetensors` is installed. If the to `False` the pipeline will *not* use

patrickvonplaten · 2023-01-27T16:59:41Z

src/diffusers/pipelines/pipeline_utils.py

+                If set to `True`, the pipeline will forcibly load the models using `safetensors` weights. If set to
+                `None` (the default). The pipeline will load using `safetensors` if the safetensors weights are
+                actually available *and* you have the library installed. If the to `False` the pipeline will *not* use
+                `safetensors` at all.


Suggested change

`safetensors` at all.

`safetensors`.

patrickvonplaten · 2023-01-27T17:00:12Z

src/diffusers/pipelines/pipeline_utils.py

@@ -505,13 +511,13 @@ def from_pretrained(cls, pretrained_model_name_or_path: Optional[Union[str, os.P

            user_agent = http_user_agent(user_agent)

-            if is_safetensors_available():
+            if use_safetensors in {None, True}:


Suggested change

if use_safetensors in {None, True}:

if use_safetensors is not False:

actually I would like to avoid the extra call to the Hub if safetensors are not installed. Can we make sure that we still check for whether safetensors are installed?

patrickvonplaten · 2023-01-27T17:00:22Z

src/diffusers/pipelines/pipeline_utils.py

                info = model_info(
                    pretrained_model_name_or_path,
                    use_auth_token=use_auth_token,
                    revision=revision,
                )
-                if is_safetensors_compatible(info):
+                if use_safetensors is True or is_safetensors_compatible(info):


Suggested change

if use_safetensors is True or is_safetensors_compatible(info):

if use_safetensors or is_safetensors_compatible(info):

github-actions · 2023-02-26T15:03:29Z

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

Narsil · 2023-02-27T10:14:14Z

Unstale

patrickvonplaten · 2023-02-27T17:38:41Z

@Narsil, I think we should merge such a PR soon, but the logic has been changed quite a bit since: #2305

Can we maybe rebase this one?

patrickvonplaten · 2023-03-10T09:55:39Z

src/diffusers/models/modeling_utils.py

@@ -463,7 +469,7 @@ def from_pretrained(cls, pretrained_model_name_or_path: Optional[Union[str, os.P

            model = load_flax_checkpoint_in_pytorch_model(model, model_file)
        else:
-            if is_safetensors_available():
+            if use_safetensors in {None, True}:


Suggested change

if use_safetensors in {None, True}:

if use_safetensors is not False:

patrickvonplaten · 2023-03-10T09:58:03Z

Still very much interested in merging this soon, but before merging this we probably need the same functionality in transformers - otherwise it doesn't make too much sense.

patrickvonplaten · 2023-03-13T20:39:38Z

huggingface/transformers#22083 is in - think we can continue with this PR :-) cc @Narsil

about which weights they use.

src/diffusers/pipelines/pipeline_utils.py

Narsil · 2023-03-15T15:56:33Z

M1 failure not linked to this PR is it ?

patrickvonplaten · 2023-03-15T17:17:28Z

Thanks a lot @Narsil - looks good to me!

sayakpaul · 2023-03-16T04:25:14Z

src/diffusers/models/modeling_utils.py

+            use_safetensors (`bool`, *optional* ):
+                If set to `True`, the pipeline will forcibly load the models from `safetensors` weights. If set to
+                `None` (the default). The pipeline will load using `safetensors` if safetensors weights are available
+                *and* if `safetensors` is installed. If the to `False` the pipeline will *not* use `safetensors`.


sayakpaul

Thanks!

Cool test cases.

williamberman · 2023-03-16T05:33:43Z

src/diffusers/loaders.py

-            if (is_safetensors_available() and weight_name is None) or (
+            if (use_safetensors is not False and weight_name is None) or (


hmm, is not False is pretty confusing. With this and the above line,

use_safetensors = kwargs.pop("use_safetensors", None if is_safetensors_available() else False)

I'm a bit confused about what the default behavior of the function is supposed to be

Ah ok I see the comment later on in the PR. Could we change the code (here and in the other places where we take use_safetensors as an argument with a default) to approximately

use_safetensors = kwargs.pop("use_safetensors", None) if use_safetensors is None: use_safetensors = is_safetensors_available() if use_safetensors and not is_safetensors_available(): raise ValueError("`use_safetensors`=True but safetensors is not installed. Please install safetensors with `pip install safetenstors") # in the rest of the function just use `use_safetensors` as a boolean

The rule of thumb here is that values which can be booleans or None can be hard to read/code with, especially when used in conditionals

But that's not it here.

If safetensors is installed, but the weights do not exist on the remote, we don't want to fail, we want to use seamlessly the PT weights.

That's why it gets more complex.

False -> Use only PT (easy)
None -> Use safetensors if both available on system and remote (which we don't know when entering the function and without checking the remote)
True -> USe only safetensors (easy)

Ah ok, I think the same logic still mostly applies

We can move the error check first and add another boolean for the case when we are allowed to fallback to non-safetensors

use_safetensors = kwargs.pop("use_safetensors", None) if use_safetensors and not is_safetensors_available(): raise ValueError("`use_safetensors`=True but safetensors is not installed. Please install safetensors with `pip install safetenstors") if use_safetensors is None: use_safetensors = is_safetensors_available() allow_pickle = True # in the rest of the function just use `use_safetensors` as a boolean # somewhere else if use_safetensors and not_safetensors_compatible(...): if not allow_pickle: raise ValueError("... pickle not allowed ...") # do the pickle loading

Is this correct or am I still missing something?

2 booleans work, Not sure it's easier to read ( I to avoid variable that create nonsensical combos, for instance allow_pickle=True and use_safetensors=False doesn't make sense)

False
None
True

Seems easier to reason about, but indeed creates some unpythoness.

I'll move to 2 booleans if you think it's clearer

t'would be much appreciated :)

williamberman · 2023-03-16T05:46:14Z

src/diffusers/loaders.py

-                except EnvironmentError:
+                except EnvironmentError as e:
+                    if use_safetensors is True:
+                        raise e


It's potentially not the job of this PR because EnvironmentError was here before, but EnvironmentError is a very broad error e.g. anything io related. Can we add some docs on why it's ok to catch it here if we're using safe tensors?

I can add some comment, but I don't know why it's EnvironmentError, I'm guessing it could be IOError since it's a file missing error, but maybe we also want to catch other types of "missing error".

If it's an IOError because a file is missing, should we just explicitly check for the file not being present? cc @patrickvonplaten I think you committed this, do you have context?

williamberman · 2023-03-16T05:58:58Z

tests/models/test_models_unet_2d_condition.py

+            with self.assertRaises(EnvironmentError):
+                new_model.load_attn_procs(tmpdirname, use_safetensors=True)


Super nit: Similar as above, EnvironmentError is really broad. Should we be checking against a specific error class or error message for this test case that safetensors could not be used?

tests/test_pipelines.py

williamberman · 2023-03-16T06:33:24Z

Love this and think it makes a lot of sense :) Just a few nits on errors and how we handle default cases

Co-authored-by: Will Berman <[email protected]>

Narsil · 2023-03-16T07:47:47Z

@williamberman adressed all your comments I think.

williamberman

love it!

williamberman · 2023-03-16T08:28:46Z

src/diffusers/loaders.py

-                except EnvironmentError:
+                except IOError as e:
+                    if not allow_pickle:
+                        raise e


@patrickvonplaten before you merge can you double check this error change is ok?

patrickvonplaten · 2023-03-16T14:57:23Z

Tbh, I thought it was better before, also because the code was more in line how we've merged it to transformes: huggingface/transformers#22083 and also because I don't think we should introduce new mental concepts like allow_pickle that the user has to understand in order to read the code (many people have no idea what pickle is and we're never talking about it in the code).

Ok to merge now

…ingface#2123) * Adding `use_safetensors` argument to give more control to users about which weights they use. * Doc style. * Rebased (not functional). * Rebased and functional with tests. * Style. * Apply suggestions from code review * Style. * Addressing comments. * Update tests/test_pipelines.py Co-authored-by: Will Berman <[email protected]> * Black ??? --------- Co-authored-by: Patrick von Platen <[email protected]> Co-authored-by: Will Berman <[email protected]>

Narsil mentioned this pull request Jan 26, 2023

Adding some safetensors docs. #2122

Merged

Narsil requested a review from patrickvonplaten January 26, 2023 16:25

patrickvonplaten reviewed Jan 27, 2023

View reviewed changes

patrickvonplaten mentioned this pull request Jan 31, 2023

Is there a way when calling .from_pretrained() to download either the safetensor or original instead of both? #1873

Closed

github-actions bot added the stale Issues that haven't received updates label Feb 26, 2023

patrickvonplaten mentioned this pull request Mar 10, 2023

[From pretrained] Speed-up loading from cache #2515

Merged

patrickvonplaten reviewed Mar 10, 2023

View reviewed changes

patrickvonplaten mentioned this pull request Mar 10, 2023

[Safetensors] Add explicit flag to from pretrained huggingface/transformers#22083

Merged

Narsil added 3 commits March 14, 2023 13:27

Adding use_safetensors argument to give more control to users

864c00e

about which weights they use.

Doc style.

b2236b8

Rebased (not functional).

b2ef1de

Merge branch 'main' into add_use_safetensors

dd066f9

patrickvonplaten reviewed Mar 14, 2023

View reviewed changes

src/diffusers/pipelines/pipeline_utils.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Mar 14, 2023

View reviewed changes

src/diffusers/pipelines/pipeline_utils.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Mar 14, 2023

View reviewed changes

src/diffusers/pipelines/pipeline_utils.py Outdated Show resolved Hide resolved

patrickvonplaten and others added 2 commits March 14, 2023 21:28

Apply suggestions from code review

756063f

Style.

f86261b

patrickvonplaten requested review from williamberman and sayakpaul March 15, 2023 17:17

sayakpaul reviewed Mar 16, 2023

View reviewed changes

sayakpaul approved these changes Mar 16, 2023

View reviewed changes

williamberman reviewed Mar 16, 2023

View reviewed changes

tests/test_pipelines.py Outdated Show resolved Hide resolved

Narsil and others added 4 commits March 16, 2023 07:48

Addressing comments.

3a2899f

Update tests/test_pipelines.py

9d35f16

Co-authored-by: Will Berman <[email protected]>

Merge branch 'main' into add_use_safetensors

09a2ae1

Black ???

89dadb7

williamberman approved these changes Mar 16, 2023

View reviewed changes

williamberman reviewed Mar 16, 2023

View reviewed changes

patrickvonplaten merged commit d9227cf into huggingface:main Mar 16, 2023

Narsil deleted the add_use_safetensors branch March 16, 2023 15:00

	use_safetensors (`bool`, optional, defaults to `None`):
	use_safetensors (`bool`, optional):

	If set to `True`, the pipeline will forcibly load the models using `safetensors` weights. If set to
	If set to `True`, the pipeline will forcibly load the model from `safetensors` weights. If set to

	`None` (the default). The pipeline will load using `safetensors` if the safetensors weights are
	`None` (the default). The pipeline will load using `safetensors` if safetensors weights are

	actually available and you have the library installed. If the to `False` the pipeline will not use
	available and if `safetensors` is installed. If set to `False` the pipeline will not use

	`None` (the default). The pipeline will load using `safetensors` if the safetensors weights are
	`None` (the default), the pipeline will be loaded from `safetensors` if the safetensors weights are

	if use_safetensors in {None, True}:
	if use_safetensors is not False:

	if use_safetensors is True or is_safetensors_compatible(info):
	if use_safetensors or is_safetensors_compatible(info):

		if (is_safetensors_available() and weight_name is None) or (
		if (use_safetensors is not False and weight_name is None) or (

		with self.assertRaises(EnvironmentError):
		new_model.load_attn_procs(tmpdirname, use_safetensors=True)

Adding use_safetensors argument to give more control to users #2123

Adding use_safetensors argument to give more control to users #2123

Uh oh!

Conversation

Narsil commented Jan 26, 2023

Uh oh!

HuggingFaceDocBuilderDev commented Jan 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

patrickvonplaten commented Jan 26, 2023

Uh oh!

sgugger commented Jan 26, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten Jan 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 26, 2023

Uh oh!

Narsil commented Feb 27, 2023

Uh oh!

patrickvonplaten commented Feb 27, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten commented Mar 10, 2023

Uh oh!

patrickvonplaten commented Mar 13, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Narsil commented Mar 15, 2023

Uh oh!

patrickvonplaten commented Mar 15, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Adding `use_safetensors` argument to give more control to users #2123

Adding `use_safetensors` argument to give more control to users #2123

HuggingFaceDocBuilderDev commented Jan 26, 2023 •

edited

Loading

patrickvonplaten Jan 27, 2023 •

edited

Loading

williamberman Mar 16, 2023 •

edited

Loading

williamberman Mar 16, 2023 •

edited

Loading

patrickvonplaten commented Mar 16, 2023 •

edited

Loading