Controlnet 1.1 Shuffle should only be used on CFG side, even when not in guess mode

[hafriedlander]

Describe the bug

Most ControlNet models should run on both conditional and unconditional sides, except in guess mode where they only run on the conditional side.

Controlnet 1.1 Shuffle model should only run on the conditional side all the time, whether in guess mode or not. It looks like the current controlnet pipeline doesn't handle that.

Reproduction

Original code:

https://github.com/lllyasviel/ControlNet-v1-1-nightly/blob/main/gradio_shuffle.py#L52

(compare to https://github.com/lllyasviel/ControlNet-v1-1-nightly/blob/main/gradio_seg.py#L67)

and note at bottom of https://github.com/lllyasviel/ControlNet-v1-1-nightly#controlnet-11-shuffle

Logs

No response

System Info

All systems, Diffusers 0.16.0

[takuma104]

Hi @hafriedlander , thanks for opening this issue. I was just working on the Global Average Pooling support for the Shuffle in ControlNetv1.1, but I had missed this point. Simply setting guess_mode to True and running it won't produce the same results as the original because the control_scales are not the same as guess_mode.

Since both this support and the Global Average Pooling support are unique features of this shuffle model, I'm thinking of modifying it so that they are set in config.json and automatically configured when the model is loaded, rather than having the user set them during use.

[takuma104]

@patrickvonplaten I just noticed #3121. I'm thinking a plan to address this Issue, what do you think?

1. Add the following three keys to the ControlNet model's config.json. If these keys are not present, the default values will be used. These keys are currently shuffle-specific, and I think the config.json will need to be edited directly.

• guess_mode: bool (default: False, suffle: True)
• guess_mode_scale_from: float (default: 0.1, shuffle: 1.0)
• guess_mode_scale_to: float (default: 1.0, shuffle: 1.0)

2. ControlNetModel's guess_mode will adopt the OR value of this config value and the one specified by the user.

3. The guess_mode in __call__() of StableDiffusionControlNetPipe will adopt the OR value of ControlNet's guess_mode.

I'm thinking of this policy based on the following two points:

• It's better users can use without being too aware of them.
• I think this function and global_pool_conditions may not always be used together in the future models.

[patrickvonplaten]

A nice catch @hafriedlander and @takuma104

Hmm, I'm not too keen on adding guess_mode to the config as IMO it's really something that should be set by runtime. Just to better understand the issue, couldn't we solve it by just writing in the docs that shuffle should always be run in guess_mode?

E.g. we could just update this:
https://huggingface.co/lllyasviel/control_v11e_sd15_shuffle#example
no?

[takuma104]

@patrickvonplaten If simply setting it to guess_mode would be enough, that would be the least labor-intensive, but looking at the original implementation, it seems to behave similarly to guess_mode but not exactly the same. The part that sets the scale for each block specific to guess_mode from 0.1 to 1.0 is not implemented and seems to be done intentionally.

For example, this part in seg, which is not shuffle:
https://github.com/lllyasviel/ControlNet-v1-1-nightly/blob/main/gradio_seg.py#L74

In shuffle, it's fixed at 1.0.
https://github.com/lllyasviel/ControlNet-v1-1-nightly/blob/main/gradio_shuffle.py#L59

The concern is what to do with this part. It's a matter of how much compatibility we should aim for.

---

After writing the proposal above, I was reading the code of mikubill/sd-webui-controlnet, where the original author has been actively committing recently. When global_average_pooling is specified to the model, it performs guess_mode equivalent (however, the scale is fixed at 1.0 as mentioned above) implicitly.

https://github.com/Mikubill/sd-webui-controlnet/blob/main/scripts/hook.py#L214

So, how about simplifying the proposal a bit and just doing the following?

• If global_pool_conditions is specified in ControlNetModel's config, perform guess_mode equivalent processing in the __call()__ function of StableDiffusionControlNetPipe.

[patrickvonplaten]

I like the idea of leveraging global_pool_conditions - sounds good to me to use global_pool_conditions as a condition to change the scale

[hafriedlander]

Relevant is the "Control Mode (previously called Guess Mode)" section on https://github.com/Mikubill/sd-webui-controlnet#new-features-in-a1111-controlnet-extension-11 - although there's no shuffle mode specific details.

[hafriedlander]

The important code from that repo is https://github.com/Mikubill/sd-webui-controlnet/blob/main/scripts/controlnet.py#L1256 (where the control arguments are determined for a model) and https://github.com/Mikubill/sd-webui-controlnet/blob/main/scripts/hook.py#L211 (where those control arguments affect cfg/ucfg and weights)

I'm pretty sure is_adapter means T2I adapter, so can be ignored for this.

[takuma104]

@hafriedlander @patrickvonplaten I just opened a draft PR #3340. If you have any comments, please let me know.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.