Skip to content
This repository was archived by the owner on Nov 27, 2024. It is now read-only.

Add ControlNet Support #88

Merged
merged 12 commits into from
Jan 5, 2024
Merged

Add ControlNet Support #88

merged 12 commits into from
Jan 5, 2024

Conversation

saddam213
Copy link
Member

@saddam213 saddam213 commented Jan 3, 2024

Add ControlNet support to OnnxStack.StabeDiffusion

This has need a highly requested feature, so time to get it underway.

Implementation wise it seems straight forward, we just have an extra model file, this being the control model, depth, edge etc

We will run the Control model first then feed the result into the Unet of the defined stable diffusion model

Figuring out the inputs/outputs of the ControlNet model and how to pass that model down as an input are the main bits to sort out, the rest should be fairly easy.

Basic Outline

  • Pipeline - Add ControlNet pipeline
  • Diffusers - Image, Inpaint, Video
  • ModelSet - Pass a "Control ModelSet" with the main ModelSet somehow? submodule time?
  • Feature Extractors - Canny, HardEdge, Depth, OpenPose
  • Basic UI integration

https://github.com/lllyasviel/ControlNet

@saddam213 saddam213 added the enhancement New feature or request label Jan 3, 2024
@saddam213
Copy link
Member Author

Success, ControlNet prototype is starting to spit out results :)

Input Output Output Output
Capture image-39377943 image-298958035 image-958323093
image-1518205137 image-1880221665 image-1906936111

Long way to go, but a good start... :)

@saddam213
Copy link
Member Author

ControlNet Scribble
image

@saddam213
Copy link
Member Author

image

@saddam213
Copy link
Member Author

ImageService to process input images is complete, we can now process Canny, HardEdge(HED) and Depth inputs

Input Canny HardEdge Depth
image-28809399 Canny Hed Depth

@saddam213 saddam213 merged commit fc9ef24 into master Jan 5, 2024
@saddam213 saddam213 deleted the ControlNet branch January 5, 2024 02:45
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant