---
title: Normalized Low-Rank Adaptation
type: vocabulary
url: "https://www.envisioning.com/vocab/normalized-low-rank-adaptation"
summary: A LoRA variant that normalizes the down-projection matrix during training, improving convergence, stability, and forgetting at no extra cost.
year: 2026
generality: 0.40
---

# Normalized Low-Rank Adaptation

A LoRA variant that normalizes the down-projection matrix during training, improving convergence, stability, and forgetting at no extra cost.
Normalized Low-Rank Adaptation (NoRA) is a modification to LoRA, the parameter-efficient fine-tuning technique, that normalizes the "down-projection" matrix used in LoRA's low-rank update during training. Jiale Kang, Ziyin Yue, Zheng Zhan, Yangyi Huang, and Weiyang Liu introduced it in "Normalized Low-Rank Adaptation" (arXiv:2608.31036, August 2026). The authors note that because LoRA initializes its "up-projection" matrix to zero, the model's early optimization trajectory is governed almost entirely by the down-projection matrix. That matrix's scale is otherwise unregularized and left to an arbitrary random initialization. NoRA addresses this by constraining each column of the down-projection matrix to unit norm along the rank dimension throughout training, which keeps the scale of every rank direction balanced and makes the resulting update easier to reason about.

The authors also test a cheaper variant, NoRA-init, which applies the same unit-norm normalization only once at initialization rather than repeatedly throughout training. NoRA-init recovers most of the benefit of full NoRA at a lower computational cost, since normal LoRA training proceeds afterward without further intervention. Both variants preserve LoRA's core properties: the low-rank update remains linear in the input and can still be merged into the frozen base weights after training. Neither variant adds trainable parameters or inference-time computation compared to standard LoRA.

Across pretraining, supervised fine-tuning, and reinforcement learning settings, the authors report that NoRA speeds up convergence, improves final task performance, produces more stable training, and reduces catastrophic forgetting relative to standard LoRA. The change requires no additional cost and applies to the general LoRA formulation rather than a narrow use case, and the authors present it as a broadly applicable improvement to LoRA rather than a specialized variant for one setting.

---
Source: Envisioning — Technology Research Institute (https://www.envisioning.com/vocab/normalized-low-rank-adaptation)