[WIP]Vae preprocessor refactor (PR1) #3557

yiyixuxu · 2023-05-25T16:06:02Z

VaeImageProcessor.preprocess refactor

refactored VaeImageProcessor a little bit
- allow passing optional height and width argument to resize()
- add convert_to_rgb
refactored prepare_latents method for img2img pipelines so that if we pass latents directly as image input, it will not encode it again
added a test in test_pipelines_common.py to test latents as image inputs
refactored img2img pipelines that accept latents as image: controlnet img2img, stable diffusion img2img , instruct_pix2pix

HuggingFaceDocBuilderDev · 2023-05-25T16:13:38Z

The documentation is not available anymore as the PR was closed or merged.

patrickvonplaten

Nice start! Can we maybe first merge the PR that adds VAE preprocess and then merge this one? Otherwise people will see lots of deprecation warnings 😅

…h_map

Co-authored-by: Pedro Cuenca <[email protected]>

yiyixuxu · 2023-06-01T02:35:46Z

src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_upscale.py

+        else:
+            do_denormalize = [not has_nsfw for has_nsfw in has_nsfw_concept]
+
+        image = self.image_processor.postprocess(image, output_type=output_type, do_denormalize=do_denormalize)


@patrickvonplaten I refactored the 4x upscaler here (just preprocess and postprocess, not accepting latents)
however I think I changed the logic of postprocess here, i.e. if output_type ==pt, currently it will return a pytorch tensor that's unnormalized, which is inconsistent with image_processor.postprocess. Let me know if we actually intend to return pytorch tensor between [-1,1] for this pipeline though

If previously, the latent upscaler was returning unnormalized tensors, I would prefer keeping it that way to avoid any unforeseen consequences?

Maybe, we could add a flag to image_processor.postprocess to check if normalization is needed? To me, that is a cleaner and more idiomatic approach.

@yiyixuxu could we for now make sure that the output stays exactly the same. I.e. we should not change the behavior of the pipelines in any way IMO.

src/diffusers/image_processor.py

sayakpaul · 2023-06-01T03:03:19Z

src/diffusers/image_processor.py

+        """
+        Convert a PIL image or a list of PIL images to numpy image
+        """
+        if not isinstance(images, list):


Is there a need to also check if image is of type PIL.Image.Image?

sayakpaul · 2023-06-01T03:04:36Z

src/diffusers/image_processor.py

+        width, height = (
+            x - x % self.config.vae_scale_factor for x in (width, height)
+        )  # resize to integer multiple of vae_scale_factor
+        image = image.resize((width, height), resample=PIL_INTERPOLATION[self.config.resample])


sayakpaul · 2023-06-01T03:08:10Z

src/diffusers/pipelines/controlnet/pipeline_controlnet.py

+        self.image_processor = VaeImageProcessor(vae_scale_factor=self.vae_scale_factor, do_convert_rgb=True)
+        self.control_image_processor = VaeImageProcessor(
+            vae_scale_factor=self.vae_scale_factor, do_convert_rgb=True, do_normalize=False
+        )


Nice to see this coming to fruition.

sayakpaul · 2023-06-01T03:14:47Z

src/diffusers/pipelines/controlnet/pipeline_controlnet.py

+        if (
+            not image_is_pil
+            and not image_is_tensor
+            and not image_is_np
+            and not image_is_pil_list
+            and not image_is_tensor_list
+            and not image_is_np_list
+        ):


That's a lot of conditions hahaha.

sayakpaul · 2023-06-01T03:19:58Z

src/diffusers/pipelines/controlnet/pipeline_controlnet.py

-
+        image = self.control_image_processor.preprocess(image, height=height, width=width).to(dtype=torch.float32)


So, all of this logic is handled by preprocess() now? That's amazing!

sayakpaul · 2023-06-01T03:21:01Z

src/diffusers/pipelines/unidiffuser/pipeline_unidiffuser.py

+    warnings.warn(
+        "The preprocess method is deprecated and will be removed in a future version. Please"
+        " use VaeImageProcessor.preprocess instead",
+        FutureWarning,
+    )


So, our plan is to refactor this in a future PR, yes?

No this function should be fully deprecated and removed in the future

sayakpaul

Awesome!

Co-authored-by: Sayak Paul <[email protected]>

src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_latent_upscale.py

patrickvonplaten · 2023-06-02T16:34:09Z

tests/pipelines/stable_diffusion/test_stable_diffusion_instruction_pix2pix.py

@@ -199,6 +204,28 @@ def test_stable_diffusion_pix2pix_euler(self):
    def test_inference_batch_single_identical(self):
        super().test_inference_batch_single_identical(expected_max_diff=3e-3)

+    # Overwrite the default test_latents_inputs because pix2pix encode the image differently


patrickvonplaten

I think we can merge this more or less. The final missing piece seems to be this: https://github.com/huggingface/diffusers/pull/3557/files#r1214586673

Can we make sure that we don't change the output behavior in any way?

This reverts commit 0ca3473.

yiyixuxu · 2023-06-02T19:40:38Z

@patrickvonplaten
reverted changes I made to x4 upscaler and created a separate issue here #3654

patrickvonplaten · 2023-06-05T10:18:05Z

Having changed this: #3557 (comment) I think we can merge this PR 🥳

…sion_latent_upscale.py Co-authored-by: Patrick von Platen <[email protected]>

VaeImageProcessor.preprocess refactor * refactored VaeImageProcessor - allow passing optional height and width argument to resize() - add convert_to_rgb * refactored prepare_latents method for img2img pipelines so that if we pass latents directly as image input, it will not encode it again * added a test in test_pipelines_common.py to test latents as image inputs * refactored img2img pipelines that accept latents as image: - controlnet img2img, stable diffusion img2img , instruct_pix2pix --------- Co-authored-by: yiyixuxu <yixu310@gmail,com> Co-authored-by: Patrick von Platen <[email protected]> Co-authored-by: Pedro Cuenca <[email protected]> Co-authored-by: Sayak Paul <[email protected]>

yiyixuxu added 4 commits May 25, 2023 15:50

add warning and add back copy from for img2img

4b15eb4

style

7cd26ef

fix copies

d58276c

add import warning

4c75754

yiyixuxu changed the title ~~Vae preprocessor~~ Vae preprocessor refactor (PR1) May 25, 2023

update

db25d47

patrickvonplaten reviewed May 25, 2023

View reviewed changes

yiyixuxu added 3 commits May 26, 2023 21:14

handle image as latents

f779977

add a test for latents input

c9ef024

add

bc66a2a

yiyixuxu changed the title ~~Vae preprocessor refactor (PR1)~~ [WIP]Vae preprocessor refactor (PR1) May 26, 2023

yiyixuxu added 18 commits May 26, 2023 22:27

copies

bf0331d

Merge remote-tracking branch 'origin/main' into vae-preprocessor

5c597c5

add

91423d7

add

19747b7

refactor img2img test

96a2a05

refactor cycle diffusion

f7be498

refactor depth pipeline preprocess + handle np input for prepare_dept…

df73ada

…h_map

refactor latent tester

eb23e59

pix2pix0

1655b1b

add a pt_np_pil input tests to pix2pix zero

076afc8

refactor postprocess for pix2pix0 invert method

706a53e

remove the unused image parameter from pix2pix0 call method

d63c444

add test for postprocess for pix2pix0 invert

cc7fffc

alt

007deda

style

188de72

fix-copies

eae044f

instruction pix2pix

0f01f9c

fix

3a95cc6

yiyixuxu and others added 6 commits May 31, 2023 08:15

Update src/diffusers/pipelines/controlnet/pipeline_controlnet_img2img.py

90110ec

Co-authored-by: Pedro Cuenca <[email protected]>

Update src/diffusers/pipelines/controlnet/pipeline_controlnet.py

1c3457e

Co-authored-by: Pedro Cuenca <[email protected]>

add doc string, images -> image

76bc363

add types and doc strings for image input arguments that's refactored

f8a0705

alt copy

7af8a1f

refacotor 4x upscale

0ca3473

yiyixuxu commented Jun 1, 2023

View reviewed changes

refactor latent_upscaler

c822a27

sayakpaul reviewed Jun 1, 2023

View reviewed changes

src/diffusers/image_processor.py Outdated Show resolved Hide resolved

sayakpaul reviewed Jun 1, 2023

View reviewed changes

src/diffusers/image_processor.py Outdated Show resolved Hide resolved

sayakpaul reviewed Jun 1, 2023

View reviewed changes

sayakpaul approved these changes Jun 1, 2023

View reviewed changes

yiyixuxu and others added 2 commits May 31, 2023 17:52

Apply suggestions from code review

ee38476

Co-authored-by: Sayak Paul <[email protected]>

add typing

2f13a14

patrickvonplaten reviewed Jun 2, 2023

View reviewed changes

src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_latent_upscale.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Jun 2, 2023

View reviewed changes

Revert "refacotor 4x upscale"

b02ca59

This reverts commit 0ca3473.

Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffu…

9a5dd1a

…sion_latent_upscale.py Co-authored-by: Patrick von Platen <[email protected]>

yiyixuxu merged commit 5990014 into main Jun 5, 2023

patrickvonplaten deleted the vae-preprocessor branch June 6, 2023 09:20


		image = self.control_image_processor.preprocess(image, height=height, width=width).to(dtype=torch.float32)

[WIP]Vae preprocessor refactor (PR1) #3557

[WIP]Vae preprocessor refactor (PR1) #3557

Uh oh!

Conversation

yiyixuxu commented May 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented May 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

patrickvonplaten left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yiyixuxu commented Jun 2, 2023

Uh oh!

patrickvonplaten commented Jun 5, 2023

Uh oh!

Uh oh!

yiyixuxu commented May 25, 2023 •

edited

Loading

HuggingFaceDocBuilderDev commented May 25, 2023 •

edited

Loading

patrickvonplaten left a comment •

edited

Loading