Documentation for LoRAConfig.

[brynhayder]

peft/src/peft/tuners/lora/config.py

Lines 112 to 113 in 162d7e5

	initialization scaled by the LoRA rank for linear and layers. Setting the initialization to False leads to
	completely random initialization and is discouraged. Pass `'loftq'` to use LoftQ initialization. Pass

Documentation for False is not clear. Presumably 'completely random' means the arrays will be uninitialized and hence contain whatever the contents of the relevant memory locations are?

[BenjaminBossan]

To explain further: The default implementation initializes the LoRA A parameter randomly and the LoRA B parameter to zeros. This results in LoRA being an identity transform at initialization, which can help with training. When setting init_lora_weights=False, the LoRA B weight is instead also randomly initialized, resulting in a non-identity transform.

For real LoRA training, you almost never want that, which is why we discourage it. However, the weights are not initialized as random memory as in torch.empty, which seems to be what you suspected.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[brynhayder]

To explain further: The default implementation initializes the LoRA A parameter randomly and the LoRA B parameter to zeros. This results in LoRA being an identity transform at initialization, which can help with training. When setting init_lora_weights=False, the LoRA B weight is instead also randomly initialized, resulting in a non-identity transform.

For real LoRA training, you almost never want that, which is why we discourage it. However, the weights are not initialized as random memory as in torch.empty, which seems to be what you suspected.

Thanks. Is it possible to update the docs to explain what you've just said? I think this should be clear in the docs.