Unloading multiple loras: norms do not return to their original values

[christopher5106]

When unloading from multiple loras on flux pipeline, I believe that the norm layers are not restored here.

Shouldn't we have:

        if len(transformer_norm_state_dict) > 0:
            original_norm_layers_state_dict = self._load_norm_into_transformer(
                transformer_norm_state_dict,
                transformer=transformer,
                discard_original_layers=False,
            )
            if not hasattr(transformer, "_transformer_norm_layers"):
                 transformer._transformer_norm_layers = original_norm_layers_state_dict

[sayakpaul]

Should it not already take care of it?

diffusers/src/diffusers/loaders/lora_pipeline.py

Line 1894 in 464374f

transformer.load_state_dict(transformer._transformer_norm_layers, strict=False)

What am I missing?

Additionally, the following test does ensure its effectivity:

diffusers/tests/lora/test_lora_layers_flux.py

Line 661 in 28f48f4

def test_lora_unload_with_parameter_expanded_shapes(self):

[christopher5106]

Ah, is it possible to call load_lora_weights() multiple times on a pipeline to load multiple weights ? does it unload in between to restore original weights ?

[sayakpaul]

If you don’t call unload_lora_weights() it won’t be called automatically.

[christopher5106]

so in that case of multiple calls to load_lora_weights(), the attribute _transformer_norm_layers become overwritten by the norms of the previously loaded lora ?

[sayakpaul]

If you're loading Control LoRA and want to keep it with others, the norm layer values that came with the Control LoRA will remain like so. Otherwise, the effectiveness of Control LoRA won't fully be there.

Or am I misinterpreting the core use case here?