Control workflow (diffusers) vs ControlNet extension (original)

[lbeltrame]

Hello,

I've been testing a bit Control and I've found a few roadblocks towards implementing this workflow. This isn't posted as an issue because a. I might be getting things wrong and b. it is a discussion on how Control works, without saying things "it's broken" or "it doesn't work", but rather "Is Control, in its current state, able to support workflow X?"

My workflow in the original backend with ControlNet

• Input prompt
• Add control image(s)
• Set hires pass 2x with reasonable denoise
• Generate

My adapted workflow with Control

• Input prompt
• Add control image
• Generate

The effect of the resize options in the pipeline is different, or possibly I could be using it wrongly, because it results in a single pass + resize, which causes a noticeably drop in image quality at the end (the original backend does two actual passes, the diffusers backend just one), as the resize is done without following up with additional sampling steps.

An idea I've been toying with is to supply the generated image back to Control (feedback) and do a second pass manually, but this is more complex and error prone than having it in a single pipeline.

EDIT:

I've done it and in that case works as expected:

• Get the image from the first Control round
• Add it as init image as alongside the control image
• Set scale to 2x with preferred upscale in "Before"
• Set scale to 1x and None in "After"
• Set the denoising strength
• Hit Generate

So, should one want to replicate what the ControlNet extension does in the original backend, two steps are required.

Hence the questions:

• Is it desirable (by design) to have such a workflow supported? I mean it with no ill intent, mind you, I ask because Control is relatively new.
• Is there currently an alternative to providing a manual "feedback loop" like I wrote above?
• Is there actually a way and have I done things completely wrong?

Thanks.

[vladmandic]

i appreciate that you're looking how to improve instead of just stating its broken!
control module is new. and workflows are being added constantly - this is one that is high on radar.
in general, yes, it needs a more explicit control of second pass.

[lbeltrame]

I could if time permits, jot down my experiments in a gist. Next up is multi ControlNet (my first experiment didn't have more than Openpose, but I have a few with 3-4 different control images).

[lbeltrame]

Oh, and right. I forgot to write and I'll leave it here for posterity: using color correction with control images can cause weird coloring so it must be turned off(does SDXL still inch towards magenta/red like SD?).

[lbeltrame]

Vlad, a question: do you think it's worth collecting here the issues I file regarding my workflow (for actual bugs, not enhancement requests)? I think my tests have uncovered at least one which I'm currently trying to narrow down before filing.

[vladmandic]

sure, this is as good as place as any

[lbeltrame]

Hm, I'm starting to get the hang of it. I'll write down a list of things that may help others, and for some of these I'll file issues:

• Denoising strength has little impact with only a control image (I can get good images with setting at 0.01) - might be good to disable it in such a case ([Feature]: Disable denoising strength input in Control when only using control image-only workflows #2769)
• It may be worthwhile to implement the "pixel perfect" resizing approach used by ControlNet for control images (currently you can get blurriness and distorted colors, which may problematic for, say, depth maps) ([Feature]: Control: Implement "pixel perfect" resizing from ControlNet extension (original backend) #2771)
• Unlike the ControlNet extension, you can't "skip" units. If a unit is disabled, the following units are ignored ([Issue]: Disabled ControlNet units in Control stop execution of following units even if enabled #2770)
• Setting an initial size especially for t2i or control image-only workflows can be awkward (what resize method to use if there is nothing to be seen?) (needs discussion, I guess)
• Sending parameters from Process to Control doesn't copy prompt or resolution parameters (issue TBD, waiting to see if it's expected or not) .

[vladmandic]

• 1: not sure i understand, lets continue in [Feature]: Disable denoising strength input in Control when only using control image-only workflows #2769
• 2: partially done in a sense that resize-before now applies to preprocessor. and what kind of resizing you set is up to you.
• 3 fixed.
• 4 fixed.
• 5 is a known todo item, its just not implemented yet. i'm actually waiting for params to be finalized so i don't have to redo it every

[lbeltrame]

Thanks! I'll be making another round of tests in the next few days and note down how my workflow fits with the new changes.

[lbeltrame]

Ah, I forgot, would it be OK to file an enhancement issue for the two-pass workflow (as a tracking measure)?

[vladmandic]

Sure

[vladmandic]

Sending parameters from Process to Control doesn't copy prompt or resolution parameters

control now adds metadata and params parsing from prompt, image drag & drop are working
there are still some remaining params that are not parsed (as each has to be handled individually), but all the main ones should be handled.