SD-XL ControlNet released as part of Diffusers

[Alchete]

It looks like the Canny model has been released. Perhaps others to follow.
https://huggingface.co/diffusers/controlnet-sdxl-1.0

The discussion page details how someone got it to work with Comfy. Perhaps that recipe may help here?

[vladmandic]

its planned, but non trivial as adding a model is one thing, but it also needs a proper front end to be usable.

[wandrzej]

Can you elaborate, what is needed on front end to make it work with A1111? Are the inputs somehow different compared to SD1X? I thought it was more a matter of architecture

[vladmandic]

Existing controlnet extension does not work with diffusers at all, so entire front end of that extension should be redone natively instead of relying on extension.

[radry]

So since the controlnet extension doesn't work with diffusers, is it possible to run sdxl without diffusers?

[midcoastal]

Yes, use backend.ORIGINAL

[gohan2091]

Is ControlNet XL suppose to be working so far in Vlads A1111? I can't get it to work. I have the XL models Canny and Depth installed and showing up but the generation ignores the control nets.

[radry]

Aparently the controlnet extension DOES work with diffusers and SDXL, so why isn't it working in SD.Next?
Mikubill/sd-webui-controlnet#2039

[vladmandic]

Search. It's been discussed many times.

[radry]

Yeah I found this issue by you and the fix of the extension. Still when updating and enabling the extension in SD.next, it gets automatically disabled.
Mikubill/sd-webui-controlnet#2040
Mikubill/sd-webui-controlnet#2041

[gohan2091]

Yes, My controlnet is always disabled. When I retick it and restart server, it remains disabled. Only when I switch back to SD 1.5 settings does Controlnet become enabled.

[vladmandic]

Yes, exactly.

[gohan2091]

Yes, exactly.

? Can I not use ControlNet with SD XL in SD.Net?

[vladmandic]

so how do you even implement those difufsers CN pipelines if they are that much simpler? And I know they are.. I was wondering how would you add those "start from XY step" etc if this even is not implemented in diffusers I think, or is it somehow?

I'm not going into full how-to here, it would be a veeeey long writeup. You'll see the code once it's merged to dev.

[blacklig]

yup, no problem.. My point was if you like "re implemented" solution diffusers use in more advanced way or just somwhow twisted those pipelines? As I was playing with CN pipelines and they really seems like less advanced than what you normaly offer in SD Next for 1.5 CNs.

[vladmandic]

diffusers pipelines are not less advanced, they are just different. there is a lot you can via things like callback_on_step_end_tensor_inputs

[blacklig]

ah, ok. So you are basically not using diffusers pipe lines but you are calling same stuff like diffusers pipe lines are calling under the hood? Only you call it in the way you need it to CN properly work and get all those settings we know from 1.5 implementation in Next?

[vladmandic]

i am using pipelines, but i via pipeline callbacks, you can modify pipeline internals on-the-fly.

[fcabanski]

Thank goodness it's coming soon. A1111 is OK, but SDNext is much better. Once it has Controlnet for SDNext, I can get back to using it.

Vlad is a wonder man!