feat: refine loRA diffusers to flux conversion logic

[simpletrontdip]

Summary

This PR updates loRA Diffusers -> Flux conversion logic based on its original source:

• Fix the key typos for guidance_in layer keys, as in: https://github.com/huggingface/diffusers/blob/55ac421f7bb12fd00ccbef727be4dc2f3f920abb/scripts/convert_flux_to_diffusers.py#L103-L115
• Add AdaLN layer from norm_out with shift scale swapping, as in : https://github.com/huggingface/diffusers/blob/55ac421f7bb12fd00ccbef727be4dc2f3f920abb/scripts/convert_flux_to_diffusers.py#L263-L268

Related Issues / Discussions

I couldn't load Hyper-FLUX.1-dev-Nsteps-lora.safetensors from https://huggingface.co/ByteDance/Hyper-SD via InvokeUI.

QA Instructions

1. Install Hyper-FLUX 8 steps, type ByteDance/Hyper-SD in the search box.
2. Load and run it with with Flux.dev flow, (to see it fail)
3. Apply the changes
4. Check the output

Merge Plan

Apply the change only, it's a small one

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable)
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

[RyanJDick]

Thanks for digging into this!

To get it merged, we'll need to:

1. Fix the shift/scale transformation.
2. Add a unit test in test_flux_diffusers_lora_conversion_utils.py for this new LoRA format. See the other tests in that file for reference.

[RyanJDick]

This doesn't look right to me. If I'm understanding correctly, in the case of a vanilla LoRA layer, we should only be flipping one of the LoRA components.

The required transformation would be a bit more involved for other LoRA variants (LoHA, LoKR, etc.), so I'm fine with only supporting vanilla LoRAs. But, we should assert that the result of any_lora_layer_from_state_dict() is a LoRALayer.

[simpletrontdip]

Hi @RyanJDick thanks for spending time 👯
I have to confess, it is more complex than I expected, sorry for not asking the team before hand.

As my understanding

# for normal LoRA layer
delta_W = up @ down
W = W + delta_W

# for AdaLN in diffusers
W_prime = swap_shift_scale(W)
delta_W_prime = swap_shift_scale(delta_W)

# => We may need to add a custom LoRA layer to swap them in `get_weight`

class AdaLN_LoRALayer(LoRALayer):
   def get_weight(self, orig_weight: torch.Tensor) -> torch.Tensor:
       '''swap shift and scale before returning real weight'''
        weight = super().get_weight(orig_weight)
        scale, shift = weight.chunk(2, dim=0)
        return torch.cat([shift, scale], dim=0)

# we need to build and return this layer in our function

What do you think?