Implement rest of the test cases (LoRA tests) #2824

aandyw · 2023-03-25T06:20:22Z

PR for issue #2789

tests/models/test_models_unet_2d_condition.py

HuggingFaceDocBuilderDev · 2023-03-25T06:26:07Z

The documentation is not available anymore as the PR was closed or merged.

aandyw · 2023-03-31T15:39:31Z

@patrickvonplaten Haven't pushed the updated code but atm I'm thinking that we add another conditional statement to:

if name.startswith("mid_block"):
      hidden_size = model.config.block_out_channels[-1]
  elif name.startswith("up_blocks"):
      block_id = int(name[len("up_blocks.")])
      hidden_size = list(reversed(model.config.block_out_channels))[block_id]
  elif name.startswith("down_blocks"):
      block_id = int(name[len("down_blocks.")])
      hidden_size = model.config.block_out_channels[block_id]

for name.startswith("transformer_in"). However, I'm not quite sure how to deal with these initial layers. I've tried ignoring them but it seems like model.set_attn_processor(lora_attn_procs) requires lora_attn_procs to contain something for the transformer_in layers.

tests/models/test_models_unet_3d_condition.py

patrickvonplaten

Looks very cool! Think your added tests are great! Went into the PR to help a bit. test_lora_processors now passes and I think you can use its logic to make the other tests as well - see: https://github.com/huggingface/diffusers/pull/2824/files#r1157208349

Let me know if you need any more help :-)

cc @sayakpaul could you also take a look here?

aandyw · 2023-04-04T16:31:22Z

Looks very cool! Think your added tests are great! Went into the PR to help a bit. test_lora_processors now passes and I think you can use its logic to make the other tests as well - see: https://github.com/huggingface/diffusers/pull/2824/files#r1157208349

Let me know if you need any more help :-)

cc @sayakpaul could you also take a look here?

Thank you for the help! I really appreciate it. Would you be able to recommend some resources that I could look into to learn more about this structure and what the transformer_in is doing specifically? I'm amazed by how fast you were able to figure out the PR.

sayakpaul · 2023-04-05T06:03:48Z

tests/models/test_models_unet_3d_condition.py

@@ -38,7 +40,10 @@
 def create_lora_layers(model):
    lora_attn_procs = {}
    for name in model.attn_processors.keys():
-        cross_attention_dim = None if name.endswith("attn1.processor") else model.config.cross_attention_dim
+        has_cross_attention = name.endswith("attn2.processor") and not (


sayakpaul · 2023-04-05T06:05:04Z

tests/models/test_models_unet_3d_condition.py

+        with torch.no_grad():
+            sample1 = model(**inputs_dict).sample
+
+        lora_attn_procs = {}


We can leverage the create_lora_layers() here and elsewhere, no?

Agree - @pie31415 could we maybe as a final todo factor out this code:

for name in model.attn_processors.keys(): has_cross_attention = name.endswith("attn2.processor") and not ( name.startswith("transformer_in") or "temp_attentions" in name.split(".") ) cross_attention_dim = model.config.cross_attention_dim if has_cross_attention else None if name.startswith("mid_block"): hidden_size = model.config.block_out_channels[-1] elif name.startswith("up_block"): block_id = int(name[len("up_blocks.")]) hidden_size = list(reversed(model.config.block_out_channels))[block_id] elif name.startswith("down_blocks"): block_id = int(name[len("down_blocks.")]) hidden_size = model.config.block_out_channels[block_id] elif name.startswith("transformer_in"): # Note that the `8 * ...` comes from: https://github.com/huggingface/diffusers/blob/7139f0e874f10b2463caa8cbd585762a309d12d6/src/diffusers/models/unet_3d_condition.py#L148 hidden_size = 8 * model.config.attention_head_dim lora_attn_procs[name] = LoRAAttnProcessor(hidden_size=hidden_size, cross_attention_dim=cross_attention_dim) with torch.no_grad(): lora_attn_procs[name].to_q_lora.up.weight += 1 lora_attn_procs[name].to_k_lora.up.weight += 1 lora_attn_procs[name].to_v_lora.up.weight += 1 lora_attn_procs[name].to_out_lora.up.weight += 1

into a create_lora_layers() as it's used three times further below? :-)

Think after this we're good for merge ❤️

sayakpaul

Thanks for taking on this one! Great work 🔥

patrickvonplaten · 2023-04-06T12:52:54Z

Looks very cool! Think your added tests are great! Went into the PR to help a bit. test_lora_processors now passes and I think you can use its logic to make the other tests as well - see: https://github.com/huggingface/diffusers/pull/2824/files#r1157208349
Let me know if you need any more help :-)
cc @sayakpaul could you also take a look here?

Thank you for the help! I really appreciate it. Would you be able to recommend some resources that I could look into to learn more about this structure and what the transformer_in is doing specifically? I'm amazed by how fast you were able to figure out the PR.

To be honest the Unet3DConditionModel is quite specific and I don't really know the reason why the authors decided to add a transformer in block 🤷

To understand what's going on, I don't have too many tips besides reading the code:

and trying to understand how they are connected. Note that we use the unet_3d_blocks.py as a file to collect many different blocks depending on the architecture while we try to have the other three files as clean as possible.

Very well done in the PR though - we're almost there!

aandyw · 2023-04-11T01:46:24Z

@patrickvonplaten @sayakpaul PR should be good to merge!

sayakpaul · 2023-04-12T08:39:37Z

tests/models/test_models_unet_2d_condition.py

-            lora_attn_procs[name] = LoRAAttnProcessor(hidden_size=hidden_size, cross_attention_dim=cross_attention_dim)
-            lora_attn_procs[name] = lora_attn_procs[name].to(model.device)
-
+        lora_attn_procs = create_lora_layers(model, mock_weights=False)


Why are we not mocking weights here?

sayakpaul · 2023-04-12T08:40:00Z

tests/models/test_models_unet_2d_condition.py

-            lora_attn_procs[name] = LoRAAttnProcessor(hidden_size=hidden_size, cross_attention_dim=cross_attention_dim)
-            lora_attn_procs[name] = lora_attn_procs[name].to(model.device)
-
+        lora_attn_procs = create_lora_layers(model, mock_weights=False)


Same question.

sayakpaul

I truly appreciate your hard work! I think this was super important.

I just have a single doubt after which we should be good to merge 🚀

patrickvonplaten · 2023-04-12T13:16:10Z

Great job @pie31415 !

* inital commit for lora test cases * help a bit with lora for 3d * fixed lora tests * replaced redundant code --------- Co-authored-by: Patrick von Platen <[email protected]> Co-authored-by: Sayak Paul <[email protected]>

inital commit for lora test cases

6cdb47c

aandyw marked this pull request as ready for review March 25, 2023 06:20

aandyw marked this pull request as draft March 25, 2023 06:20

Merge branch 'main' into lora-test-cases

2b9887f

aandyw commented Mar 25, 2023

View reviewed changes

tests/models/test_models_unet_2d_condition.py Show resolved Hide resolved

patrickvonplaten mentioned this pull request Mar 31, 2023

[UNet3DConditionModel] implement rest of the test cases (LoRA, Slow tests) #2789

Closed

help a bit with lora for 3d

5a208ce

patrickvonplaten reviewed Apr 4, 2023

View reviewed changes

tests/models/test_models_unet_3d_condition.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Apr 4, 2023

View reviewed changes

aandyw and others added 2 commits April 4, 2023 12:31

Merge branch 'huggingface:main' into lora-test-cases

fb4f2f0

fixed lora tests

c3373a1

aandyw marked this pull request as ready for review April 4, 2023 20:19

sayakpaul reviewed Apr 5, 2023

View reviewed changes

sayakpaul approved these changes Apr 5, 2023

View reviewed changes

aandyw and others added 2 commits April 10, 2023 17:41

replaced redundant code

85a4846

Merge branch 'main' into lora-test-cases

2cb926e

aandyw changed the title ~~[WIP] implement rest of the test cases (LoRA, Slow tests)~~ [WIP] implement rest of the test cases (LoRA tests) Apr 11, 2023

sayakpaul reviewed Apr 12, 2023

View reviewed changes

sayakpaul approved these changes Apr 12, 2023

View reviewed changes

Merge branch 'main' into lora-test-cases

f94705c

sayakpaul merged commit 9d7c08f into huggingface:main Apr 12, 2023

aandyw changed the title ~~[WIP] implement rest of the test cases (LoRA tests)~~ Implement rest of the test cases (LoRA tests) Jun 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Implement rest of the test cases (LoRA tests) #2824

Implement rest of the test cases (LoRA tests) #2824

Uh oh!

aandyw commented Mar 25, 2023

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Mar 25, 2023 •

edited

Loading

Uh oh!

aandyw commented Mar 31, 2023

Uh oh!

Uh oh!

patrickvonplaten left a comment

Uh oh!

aandyw commented Apr 4, 2023

Uh oh!

sayakpaul Apr 5, 2023

Uh oh!

sayakpaul Apr 5, 2023 •

edited

Loading

Uh oh!

patrickvonplaten Apr 6, 2023

Uh oh!

sayakpaul left a comment

Uh oh!

patrickvonplaten commented Apr 6, 2023

Uh oh!

aandyw commented Apr 11, 2023 •

edited

Loading

Uh oh!

sayakpaul Apr 12, 2023

Uh oh!

sayakpaul Apr 12, 2023

Uh oh!

sayakpaul left a comment

Uh oh!

patrickvonplaten commented Apr 12, 2023

Uh oh!

Uh oh!

Implement rest of the test cases (LoRA tests) #2824

Implement rest of the test cases (LoRA tests) #2824

Uh oh!

Conversation

aandyw commented Mar 25, 2023

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Mar 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aandyw commented Mar 31, 2023

Uh oh!

Uh oh!

patrickvonplaten left a comment

Choose a reason for hiding this comment

Uh oh!

aandyw commented Apr 4, 2023

Uh oh!

sayakpaul Apr 5, 2023

Choose a reason for hiding this comment

Uh oh!

sayakpaul Apr 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten Apr 6, 2023

Choose a reason for hiding this comment

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten commented Apr 6, 2023

Uh oh!

aandyw commented Apr 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sayakpaul Apr 12, 2023

Choose a reason for hiding this comment

Uh oh!

sayakpaul Apr 12, 2023

Choose a reason for hiding this comment

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten commented Apr 12, 2023

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Mar 25, 2023 •

edited

Loading

sayakpaul Apr 5, 2023 •

edited

Loading

aandyw commented Apr 11, 2023 •

edited

Loading