HDR latent control

[Tillerz]

I've seen the link with the "explanation", but that doesn't really explain the three toggle to me. Anyone tried them out and has some details on what's actually happening? :)

[vladmandic]

changelog has a link to article and actual examples

[eadnams22]

Reading the document, the code in SD.Next, and doing some tests (its not available in X/Y/Z Grid yet, unfortunately), The three options trigger specific (related) modifications during generation, but at different times during generation.

First is Clamp, then Center, then Maximize (and a less severe Center).

Manipulating colors in SD/SDXL during generation is a bit weird, as the document points out, because its not using RGB, so the way to balance it is also a bit weird, it seems.

DISCLAIMER: This is as best I understand it (and thus use it), so YMMV

(Quotes from the documents explanation for each item are in italics)

HDR Clamp

This will control the amount of nonsensical details, by pruning values that are the farthest from the mean of the distribution. It also helps in generating at higher guidance_scale.

This identifies any parts of the noise that are outliers from the average, early in generation, and changes them. This is the weirdest one to me, it does a bunch of math using the Range(Boundary) and Threshold to determine if it needs to 'change' the outlier, and by what amount. It checks the tensor value against the value of threshold * range (if its over the positive number or under the negative version of the same number), then if it needs to be changed, it knocks the original tensor in to range, while ignoring any tensors that are within that happy middle.

Think of it as creating the + and - range you want your image to be in. So I set the range to 4, with a threshold of 0.9, which means anything above 3.9 or below -3.9 will get pushed back in to that range.

NOTE: At higher guidance scales(CFG scale) the base values will have a higher difference between min and max, you can adjust accordingly

Sometimes, almost everything is already in that range (depending on your CFG scale, etc), so you'll see very little difference with it turned on or off.

HDR Center

I have two main methods of achieving this. The first one is to shrink towards the mean while normalizing the values (Which will also remove outliers) and the second is to fix when the values get biased towards some color. This also helps in generating at higher guidance_scale.

Channel Shift takes the channels data, and subtracts the average * Channel Shift. So if the Channel Shift amount is 1.5, and the channel average is 4, that channels individual tensors will be changed by 2.5

Full Shift is after that, which is a similar operation except for all channels combined in the tensor, taking the average and shifting it by the average * Full Shift value.

So it shifts the channels first, towards the average, then shifts everything towards their average too, ensuring theres no unwanted or unusual color shifts.

HDR Maximize

This is basically done by multiplying the tensors by a very small amount like 1e-5 for a few steps and to make sure that the final tensor is using the full possible range [...] before converting to RGB. Remember, in the pixel space, it's easier to reduce contrast, saturation and sharpness with intact dynamics than to increase it.

Say hello to our friend "Range" (or Boundary) again!

This modifies the the HDR Center function as well.

"Center" is what it changes the "channel shift" from above to towards the end (and it locks Full Shift at 1).

Regarding "Range", This is the big one for "maximize" it takes the channels and calculates a "normalization factor" based upon the Range, by taking the maximum value it finds in the tensor, and dividing it by the Range * 4, then it uses the result to shift the channels while staying within the given boundary. So if the max is say, 6, and the Range is set to 1.2, it would take each channel and multiply it by 0.8.

NOTE: To me, this particular step is to ensure you have the maximum available dynamic range available when you go in to another program like photoshop. If you're not going in to another program to fine-tune your levels/contrast/brightness, it may not give you the desired effect.

Conclusion

These variables all work together to sort of 'average out' the colors and lighting of a generation, and maximize the usage of them within those boundaries.

Theres no real 'right' setting, as it will vary depending upon what youre generating, and your intent/vision.

[vladmandic]

good writeup
@Aptronymist can you work with @eadnams22 and make this into a wiki article? we need better docs for sure :)

[eadnams22]

Thanks! I’m on the discord if you wanna connect there.

[Aptronymist]

Of course!

[Tillerz]

Woo, thank you so much. That makes it easier to now fiddle with the values, understanding them (more or less) what they actually do. \o/

[eadnams22]

You’re welcome! Hope it helps, didn’t mean to make it that long, but I got sucked in to figuring out what the code was actually doing, and learning how colour/luminance works with SD. 😂

[Aptronymist]

You’re welcome! Hope it helps, didn’t mean to make it that long, but I got sucked in to figuring out what the code was actually doing, and learning how colour/luminance works with SD. 😂

Read it, nice stuff! I'm going to play with it a bit myself based on your instructions and get this worked up into a wiki pages in the next day or so. Good job!

[eadnams22]

I’m flattered.

I don’t code well (I can read/edit it decently enough), but I’ve been a communicator to and from devs and clients for a long time, and read code well enough to explain it in different or more understandable terms, guess it’s something I can contribute when I am able.

Also, as someone who does production/post-production, I’ve wanted a way to get more usable image/color data in SD generated images for use in other workflows/pipelines, and so wanted to figure out these HDR tools.

I hope it eventually gets to a point that we can make 32-bit generated true HDR images, and control exposure realistically with the result.

[vladmandic]

well, if you feel writing/updating any docs, i'm not gonna say no!

[eadnams22]

Hmm, it seems like it doesn’t apply quite correctly to batch generations (using diffusers backend).

If I had to guess based upon my observations, it’s doing the tweaks for the entire combined batch generation, and not each image individually, as the images typically change when you try to do a single generation from a batch (oddly, except if it’s the first image from the batch), unless you turn off the adjustments, then it looks more like the image from the batch.

Or given that behaviour, maybe it’s only doing all of the adjustments for the first image in the batch? 🤔

Seems to be the centring that throws it off the most between batch and individual, clamping may be working correctly, or at least working in a way that is consistent for batch and single images from the same seed.

Can make a bug report if that’s preferable @vladmandic .

[vladmandic]

Hmm, it seems like it doesn’t apply quite correctly to batch generations (using diffusers backend).

I've worked on that just few days ago, it should be ok in dev branch.