ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
Long-term reading2 min readDisconfirmAI Editorial

LoRA: Low-Rank Adaptation of Large Language Models

In the field of natural language processing, adapting large pre-trained models to specific tasks typically relies on full fine-tuning.

Verified 2026-11-20 00:01 GMT+8

In the field of natural language processing, adapting large pre-trained models to specific tasks typically relies on full fine-tuning. However, as model parameter scales reach hundreds of billions (e.g., GPT-3), storing and deploying independent fine-tuned copies for each downstream task becomes extremely costly and difficult to maintain. Microsoft Research proposed LoRA (Low-Rank Adaptation) to address this efficiency bottleneck.

The core assumption of LoRA is that weight changes during adaptation to new tasks exhibit an "intrinsic low-rank" property. This method freezes the original weights of the pre-trained model and injects trainable low-rank decomposition matrices at every layer of the Transformer architecture. By optimizing only these small incremental matrices rather than all parameters, LoRA significantly reduces the number of trainable parameters. Experiments show that compared to full fine-tuning of GPT-3 175B using Adam optimization, LoRA can reduce trainable parameters by a factor of 10,000, lower GPU memory requirements by threefold, while maintaining model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3 equal to or better than full fine-tuning.

Compared to other parameter-efficient methods, LoRA avoids the inference latency introduced by adapters. Adapters insert modules by increasing network depth, which leads to significant computational overhead in online inference scenarios. In contrast, LoRA employs a simple linear design that allows trainable matrices to be merged with frozen weights during deployment, achieving efficient adaptation without increasing inference latency. Additionally, LoRA is orthogonal to many existing methods and can be combined with techniques such as prefix tuning.

For engineers and researchers seeking to customize large language models but constrained by computational resources, LoRA provides a practical solution balancing efficiency and performance. It is recommended to start with Section 4 of the paper regarding the description of low-rank parameterized update matrices to understand the mathematical principles, followed by referring to the code implementation in the appendix for practice. This work was published by Edward Hu et al. in 2021 under the title "LoRA: Low-Rank Adaptation of Large Language Models".

Sign in / Sign up
ReadingResearchRadarInvestment framework

Sources

  • arxiv.org《LoRA: Low-Rank Adaptation of Large Language Models》(2021)↗
Continue researching →