---
title: Low-Rank Adaptation
type: vocabulary
url: "https://www.envisioning.com/vocab/low-rank-adaptation"
summary: A parameter-efficient fine-tuning technique that adapts a pretrained model by training a small low-rank update instead of the full weight matrices.
year: 2021
generality: 0.60
---

# Low-Rank Adaptation

A parameter-efficient fine-tuning technique that adapts a pretrained model by training a small low-rank update instead of the full weight matrices.
Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning technique for large neural networks. Rather than updating a pretrained model's full weight matrices to adapt it to a new task, LoRA trains a small pair of low-rank matrices instead. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen at Microsoft introduced it in "LoRA: Low-Rank Adaptation of Large Language Models" (arXiv:2106.09685, June 2021).

Instead of updating a weight matrix directly during fine-tuning, LoRA freezes the original pretrained weights and injects a separate update. This update is expressed as the product of two much smaller matrices, a "down-projection" and an "up-projection," whose rank is far lower than that of the original matrix. Only these small matrices are trained. Their product is added to the frozen weights at inference time, or merged into them.

Because the trainable matrices are small, LoRA cuts the number of trainable parameters and the GPU memory needed for fine-tuning by a large factor compared to updating an entire model. The original paper reported model quality on par with full fine-tuning on several language tasks. LoRA adds no extra latency at inference if the low-rank update is merged back into the base weights. It also allows many separate low-rank adapters for different tasks to be trained cheaply and swapped in and out of the same frozen base model. The up-projection matrix is conventionally initialized to zero, so the adapter starts as a no-op and training only gradually introduces a change to the model's behavior.

LoRA became one of the most widely used techniques for customizing large language and diffusion models on limited hardware. It spawned a large family of variants addressing its limitations: QLoRA combines LoRA with quantization of the frozen base weights, and later methods change how the low-rank matrices are initialized or regularized during training, including Normalized Low-Rank Adaptation (NoRA).

---
Source: Envisioning — Technology Research Institute (https://www.envisioning.com/vocab/low-rank-adaptation)