Add Differential Diffusion to HunyuanDiT. #9040

MnCSSJ4x · 2024-08-01T13:20:16Z

What does this PR do?

Adds Differential Diffusion to HunyuanDIT.

Fixes Partially #8924(HunyuanDiT Only)

Before submitting

Did you read the contributor guideline?
Did you read our philosophy doc (important for complex PRs)?

How to test:

Gradient

import torch
from diffusers import FlowMatchEulerDiscreteScheduler
from diffusers.utils import load_image
from PIL import Image
from torchvision import transforms

from pipeline_hunyuandit_differential_img2img import (
    HunyuanDiTDifferentialImg2ImgPipeline,
)


pipe = HunyuanDiTDifferentialImg2ImgPipeline.from_pretrained(
    "Tencent-Hunyuan/HunyuanDiT-Diffusers", torch_dtype=torch.float16
).to("cuda")


source_image = load_image(
    "https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/differential/20240329211129_4024911930.png"
)
map = load_image(
    "https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/differential/gradient_mask_2.png"
)
prompt = "a green pear"
negative_prompt = "blurry"

image = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    image=source_image,
    num_inference_steps=28,
    guidance_scale=4.5,
    strength=1.0,
    map=map,
).images[0]


Gradient	Input	Output

A colab notebook demonstrating all results can be found here. Depth Maps have also been added in the same colab.

Who can review?

@a-r-r-o-w @DN6

a-r-r-o-w

Thank you for working on this, looks good to me! I think we can merge this once you add your name and contrib to the community README file. Also, seems like at place of the code, the style guide is not followed. Normally, these could be fixed with make style if it was a pipeline in core diffusers. However, since this is a community pipeline, you can run styling with:
ruff check examples/community/pipeline_hunyuandit_differential_img2img.py --fix

a-r-r-o-w

Thank you for working on this, looks good to me! I think we can merge this once you add your name and contrib to the community README file. Also, seems like at place of the code, the style guide is not followed. Normally, these could be fixed with make style if it was a pipeline in core diffusers. However, since this is a community pipeline, you can run styling with:
ruff check examples/community/pipeline_hunyuandit_differential_img2img.py --fix

MnCSSJ4x · 2024-08-03T13:12:40Z

@a-r-r-o-w I have fixed the style issues and added the details to the markdown file. If all looks good, you can go ahead and merge this request.

a-r-r-o-w · 2024-08-03T13:59:14Z

@MnCSSJ4x Could you revert all the other changes apart from adding your name and contribution to community README? If you'd like to refactor, you can do it in a separate PR as it's out of scope for this one. Please keep the changes here limited

MnCSSJ4x · 2024-08-03T14:00:50Z

@MnCSSJ4x Could you revert all the other changes apart from adding your name and contribution to community README? If you'd like to refactor, you can do it in a separate PR as it's out of scope for this one. Please keep the changes here limited

Sure. I'll try to revert in fix. I feel some tool might have auto refactored it.

This reverts commit d347de1.

This reverts commit ce7c3ff.

…into hunyuan-diff

MnCSSJ4x · 2024-08-03T14:21:18Z

@a-r-r-o-w Can you please check and let me know if it's ok now? Apologies for bothering you with such trivial issues.

a-r-r-o-w · 2024-08-03T14:26:56Z

No problem, but there are still a large number of differences. Please run git restore -s main examples/community/README.md first, and then add your name to the table, and an example of how to run Hunyuan DiffDiff. You will know the PR is ready if it contains only additions here. Currently, it contains many deletions due to additional changes

MnCSSJ4x · 2024-08-03T15:11:55Z

@a-r-r-o-w Thanks for the command. It should be resolved now.

examples/community/README.md

examples/community/pipeline_hunyuandit_differential_img2img.py

a-r-r-o-w · 2024-08-03T15:41:20Z

@MnCSSJ4x Looking good implementation-wise. The quality tests seems to be failing. Could you run make style and ensure that the tests pass locally too.

HuggingFaceDocBuilderDev · 2024-08-03T15:42:44Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

MnCSSJ4x · 2024-08-03T16:12:29Z

@MnCSSJ4x Looking good implementation-wise. The quality tests seems to be failing. Could you run make style and ensure that the tests pass locally too.

Upon running make style I get the following and am unable to track back where exactly things are going wrong or if I introduced this issues.

ruff check examples scripts src tests utils benchmarks setup.py --fix
src/diffusers/configuration_utils.py:679:16: E721 Use `is` and `is not` for type comparisons, or `isinstance()` for isinstance checks
    |
677 |             if field.name in self._flax_internal_args:
678 |                 continue
679 |             if type(field.default) == dataclasses._MISSING_TYPE:
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E721
680 |                 default_kwargs[field.name] = None
681 |             else:
    |

tests/models/test_modeling_common.py:338:20: E721 Use `is` and `is not` for type comparisons, or `isinstance()` for isinstance checks
    |
337 |         model.set_default_attn_processor()
338 |         assert all(type(proc) == AttnProcessorNPU for proc in model.attn_processors.values())
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E721
339 |         with torch.no_grad():
340 |             if self.forward_requires_fresh_args:
    |

tests/models/test_modeling_common.py:346:20: E721 Use `is` and `is not` for type comparisons, or `isinstance()` for isinstance checks
    |
345 |         model.enable_npu_flash_attention()
346 |         assert all(type(proc) == AttnProcessorNPU for proc in model.attn_processors.values())
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E721
347 |         with torch.no_grad():
348 |             if self.forward_requires_fresh_args:
    |

tests/models/test_modeling_common.py:354:20: E721 Use `is` and `is not` for type comparisons, or `isinstance()` for isinstance checks
    |
353 |         model.set_attn_processor(AttnProcessorNPU())
354 |         assert all(type(proc) == AttnProcessorNPU for proc in model.attn_processors.values())
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E721
355 |         with torch.no_grad():
356 |             if self.forward_requires_fresh_args:
    |

tests/models/test_modeling_common.py:389:20: E721 Use `is` and `is not` for type comparisons, or `isinstance()` for isinstance checks
    |
388 |         model.set_default_attn_processor()
389 |         assert all(type(proc) == AttnProcessor for proc in model.attn_processors.values())
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^ E721
390 |         with torch.no_grad():
391 |             if self.forward_requires_fresh_args:
    |

tests/models/test_modeling_common.py:397:20: E721 Use `is` and `is not` for type comparisons, or `isinstance()` for isinstance checks
    |
396 |         model.enable_xformers_memory_efficient_attention()
397 |         assert all(type(proc) == XFormersAttnProcessor for proc in model.attn_processors.values())
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E721
398 |         with torch.no_grad():
399 |             if self.forward_requires_fresh_args:
    |

tests/models/test_modeling_common.py:405:20: E721 Use `is` and `is not` for type comparisons, or `isinstance()` for isinstance checks
    |
404 |         model.set_attn_processor(XFormersAttnProcessor())
405 |         assert all(type(proc) == XFormersAttnProcessor for proc in model.attn_processors.values())
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E721
406 |         with torch.no_grad():
407 |             if self.forward_requires_fresh_args:
    |

tests/models/test_modeling_common.py:433:20: E721 Use `is` and `is not` for type comparisons, or `isinstance()` for isinstance checks
    |
431 |             return
432 | 
433 |         assert all(type(proc) == AttnProcessor2_0 for proc in model.attn_processors.values())
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E721
434 |         with torch.no_grad():
435 |             if self.forward_requires_fresh_args:
    |

tests/models/test_modeling_common.py:441:20: E721 Use `is` and `is not` for type comparisons, or `isinstance()` for isinstance checks
    |
440 |         model.set_default_attn_processor()
441 |         assert all(type(proc) == AttnProcessor for proc in model.attn_processors.values())
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^ E721
442 |         with torch.no_grad():
443 |             if self.forward_requires_fresh_args:
    |

tests/models/test_modeling_common.py:449:20: E721 Use `is` and `is not` for type comparisons, or `isinstance()` for isinstance checks
    |
448 |         model.set_attn_processor(AttnProcessor2_0())
449 |         assert all(type(proc) == AttnProcessor2_0 for proc in model.attn_processors.values())
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E721
450 |         with torch.no_grad():
451 |             if self.forward_requires_fresh_args:
    |

tests/models/test_modeling_common.py:457:20: E721 Use `is` and `is not` for type comparisons, or `isinstance()` for isinstance checks
    |
456 |         model.set_attn_processor(AttnProcessor())
457 |         assert all(type(proc) == AttnProcessor for proc in model.attn_processors.values())
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^ E721
458 |         with torch.no_grad():
459 |             if self.forward_requires_fresh_args:
    |

tests/pipelines/controlnet/test_controlnet_sdxl.py:1022:16: E721 Use `is` and `is not` for type comparisons, or `isinstance()` for isinstance checks
     |
1021 |         controlnet = ControlNetModel.from_unet(unet, conditioning_channels=4)
1022 |         assert type(controlnet.mid_block) == UNetMidBlock2D
     |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E721
1023 |         assert controlnet.conditioning_channels == 4
     |

tests/pipelines/test_pipelines_common.py:777:21: E721 Use `is` and `is not` for type comparisons, or `isinstance()` for isinstance checks
    |
775 |             if hasattr(component, "attn_processors"):
776 |                 assert all(
777 |                     type(proc) == AttnProcessor for proc in component.attn_processors.values()
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^ E721
778 |                 ), "`from_pipe` changed the attention processor in original pipeline."
    |

tests/schedulers/test_schedulers.py:827:16: E721 Use `is` and `is not` for type comparisons, or `isinstance()` for isinstance checks
    |
825 |         scheduler_loaded = DDIMScheduler.from_pretrained(f"{USER}/{self.repo_id}")
826 | 
827 |         assert type(scheduler) == type(scheduler_loaded)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E721
828 | 
829 |         # Reset repo
    |

tests/schedulers/test_schedulers.py:838:16: E721 Use `is` and `is not` for type comparisons, or `isinstance()` for isinstance checks
    |
836 |         scheduler_loaded = DDIMScheduler.from_pretrained(f"{USER}/{self.repo_id}")
837 | 
838 |         assert type(scheduler) == type(scheduler_loaded)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E721
839 | 
840 |         # Reset repo
    |

tests/schedulers/test_schedulers.py:854:16: E721 Use `is` and `is not` for type comparisons, or `isinstance()` for isinstance checks
    |
852 |         scheduler_loaded = DDIMScheduler.from_pretrained(self.org_repo_id)
853 | 
854 |         assert type(scheduler) == type(scheduler_loaded)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E721
855 | 
856 |         # Reset repo
    |

tests/schedulers/test_schedulers.py:865:16: E721 Use `is` and `is not` for type comparisons, or `isinstance()` for isinstance checks
    |
863 |         scheduler_loaded = DDIMScheduler.from_pretrained(self.org_repo_id)
864 | 
865 |         assert type(scheduler) == type(scheduler_loaded)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E721
866 | 
867 |         # Reset repo
    |

Found 17 errors.
make: *** [style] Error 1

a-r-r-o-w · 2024-08-03T16:14:35Z

Is your ruff version same as the one in our setup.py? I remember seeing something in the past due to incompatible ruff versions

MnCSSJ4x · 2024-08-05T09:28:37Z

Is your ruff version same as the one in our setup.py? I remember seeing something in the past due to incompatible ruff versions

Hi, yes the version was different. Fixed it and ran the command make style, I think the formatting was fixed and I pushed the same.

However, Got some error text

ruff check examples scripts src tests utils benchmarks setup.py --fix
ruff format examples scripts src tests utils benchmarks setup.py
1 file reformatted, 1036 files left unchanged
doc-builder style src/diffusers docs/source --max_len 119
make: doc-builder: No such file or directory
make: *** [style] Error 1

a-r-r-o-w

Thank you for your contribution and bearing with our reviews! This is a very strong good-first-issue finish 🎉

* Add Differential Pipeline. * Fix Styling Issue using ruff -fix * Add details to Contributing.md * Revert "Fix Styling Issue using ruff -fix" This reverts commit d347de1. * Revert "Revert "Fix Styling Issue using ruff -fix"" This reverts commit ce7c3ff. * Revert README changes * Restore README.md * Update README.md * Resolved Comments: * Fix Readme based on review * Fix formatting after make style --------- Co-authored-by: Aryan <[email protected]>

Add Differential Pipeline.

5621fb3

MnCSSJ4x changed the title ~~Add Differential Pipeline.~~ Add Differential Diffusion to HunyuanDiT. Aug 1, 2024

MnCSSJ4x marked this pull request as ready for review August 1, 2024 13:21

MnCSSJ4x mentioned this pull request Aug 1, 2024

Adding Differential Diffusion to HunyuanDiT #8992

Closed

2 tasks

a-r-r-o-w approved these changes Aug 2, 2024

View reviewed changes

MnCSSJ4x and others added 3 commits August 3, 2024 18:17

Fix Styling Issue using ruff -fix

d347de1

Add details to Contributing.md

78deb4f

Merge branch 'main' into hunyuan-diff

0b51f4a

MnCSSJ4x added 4 commits August 3, 2024 19:42

Revert "Fix Styling Issue using ruff -fix"

ce7c3ff

This reverts commit d347de1.

Revert "Revert "Fix Styling Issue using ruff -fix""

11d764d

This reverts commit ce7c3ff.

Merge branch 'hunyuan-diff' of https://github.com/MnCSSJ4x/diffusers …

073304e

…into hunyuan-diff

Revert README changes

ba05675

MnCSSJ4x and others added 2 commits August 3, 2024 20:04

Restore README.md

8a64bd0

Update README.md

0f73cca

a-r-r-o-w reviewed Aug 3, 2024

View reviewed changes

examples/community/README.md Outdated Show resolved Hide resolved

examples/community/README.md Outdated Show resolved Hide resolved

examples/community/pipeline_hunyuandit_differential_img2img.py Show resolved Hide resolved

MnCSSJ4x added 2 commits August 3, 2024 21:24

Resolved Comments:

426915b

Fix Readme based on review

361a696

Fix formatting after make style

3a2c3d4

Merge branch 'main' into hunyuan-diff

a571a66

a-r-r-o-w approved these changes Aug 8, 2024

View reviewed changes

a-r-r-o-w merged commit 1fcb811 into huggingface:main Aug 8, 2024
8 checks passed

Add Differential Diffusion to HunyuanDiT. #9040

Add Differential Diffusion to HunyuanDiT. #9040

Uh oh!

Conversation

MnCSSJ4x commented Aug 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Before submitting

Gradient

Who can review?

Uh oh!

a-r-r-o-w left a comment

Choose a reason for hiding this comment

Uh oh!

a-r-r-o-w left a comment

Choose a reason for hiding this comment

Uh oh!

MnCSSJ4x commented Aug 3, 2024

Uh oh!

a-r-r-o-w commented Aug 3, 2024

Uh oh!

MnCSSJ4x commented Aug 3, 2024

Uh oh!

MnCSSJ4x commented Aug 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

a-r-r-o-w commented Aug 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MnCSSJ4x commented Aug 3, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

a-r-r-o-w commented Aug 3, 2024

Uh oh!

HuggingFaceDocBuilderDev commented Aug 3, 2024

Uh oh!

MnCSSJ4x commented Aug 3, 2024

Uh oh!

a-r-r-o-w commented Aug 3, 2024

Uh oh!

MnCSSJ4x commented Aug 5, 2024

Uh oh!

a-r-r-o-w left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

MnCSSJ4x commented Aug 1, 2024 •

edited

Loading

MnCSSJ4x commented Aug 3, 2024 •

edited

Loading

a-r-r-o-w commented Aug 3, 2024 •

edited

Loading