Home / Curated / Algorithms / LoRA-FA

LoRA-FA

Algorithms AIMech:LoRAFA DRAFT

Aliases: LoRA with Frozen-A, Memory-Efficient Low-Rank Adaptation with Frozen Projection-Down Matrix, LoRA-FA Parameter-Efficient Fine-Tuning

A memory-efficient low-rank adaptation algorithm that freezes the randomly initialized down-projection matrix A and updates only the zero-initialized up-projection matrix B using a precomputed Gram-inverse gradient correction (r/alpha)^2 (A^T A)^{-1} g^B_{LoRA-FA}, eliminating the need to cache full-dimension layer input activations X during the forward pass.

Identity Notes & Concept Scope Distinct from AIMech:LowRankAdaptation (LoRA), which updates both A in R^{d_{in} x r} and B in R^{r x d_{out}} and must retain the full input activation tensor X in R^{b x s x d_{in}} in GPU memory to compute dL/dA = (alpha/r) X^T (dL/dY) B^T during backpropagation, and from AIMech:LoraPlusEfficientLowRankAdaptationWithAsymmetricLearningRates (LoRA+), which still updates both A and B with asymmetric learning rates. LoRA-FA (Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li, arXiv:2308.03303) proves that stacked linear low-rank updates collapse to single-layer linear regression within the column space of A_0 (Theorem 3.1) and derives the closed-form gradient correction g^B = (r/alpha)^2 (A^T A)^{-1} g^B_{LoRA-FA} (Theorem 4.1) that minimizes the Frobenius norm discrepancy ||(alpha/r) A g^B - g||_F^2 against the full-fine-tuning gradient g while guaranteeing dL <= 0. Because A is frozen, only the r-dimensional intermediate activation X A in R^{b x s x r} (2bsr bytes instead of 2bsd + 2bsr bytes) is stored for backpropagation, and K = (A^T A)^{-1} in R^{r x r} is precomputed once at initialization.

Claims & Grounded Evidence

4 Claims

Mechanism #lorafa-frozen-down-projection-activation-memory-elimination

By freezing the base weights W_0 and the down-projection matrix A in R^{d_{in} x r}, LoRA-FA avoids computing gradients with respect to A and therefore discards the full input activation tensor X in R^{b x s x d_{in}}, caching only the low-rank intermediate activation X A in R^{b x s x r} (reducing adapter activation memory from 2bsd + 2bsr bytes to 2bsr bytes per linear layer).

supports Source: LoRA-FA: Memory-Efficient Low-Rank Adaptation for Large Language Models Fine-Tuning · Locator: Table 1, Section 5 (LoRA-FA is more memory efficient) & Appendix C.3, pp. 3, 5, 20

Mechanism #lorafa-closed-form-gram-inverse-gradient-correction

To minimize the Frobenius discrepancy ||(alpha/r) A g^B - g||_F^2 between the induced low-rank weight gradient and the full gradient g while ensuring non-increasing loss dL <= 0, LoRA-FA precomputes K = (A^T A)^{-1} in R^{r x r} once at initialization and transforms the raw gradient g^B_{LoRA-FA} into g^B = (r/alpha)^2 (A^T A)^{-1} g^B_{LoRA-FA} before updating AdamW moments.

supports Source: LoRA-FA: Memory-Efficient Low-Rank Adaptation for Large Language Models Fine-Tuning · Locator: Section 4 (Theorem 4.1), Appendix A.2 (Equations 15-27) & Appendix B (Algorithm 1), pp. 4, 16-19

Application #lorafa-application-glue-gsm8k-humaneval-and-long-context-sft

Evaluated on RoBERTa-base/large (GLUE), LLaMA2-7B, LLaMA3-8B, Qwen3-8B, and DeepSeek-V2-Lite across GSM8K, HumanEval, and MT-Bench, LoRA-FA matches or outperforms LoRA, DoRA, and LoRA-Pro while cutting peak training memory by over 27 GB on LLaMA2-7B and sustaining higher Model FLOPs Utilization (MFU).

supports Source: LoRA-FA: Memory-Efficient Low-Rank Adaptation for Large Language Models Fine-Tuning · Locator: Section 6 (Experiments), Tables 2-4 & Figure 2, pp. 6-8

Limitation #lorafa-limitation-fixed-column-subspace-and-svd-init-blind-spot

Because the induced weight update Delta_W = -(alpha/r) P_A g is orthogonally projected onto the fixed r-dimensional column space of the frozen initial matrix A, any gradient component orthogonal to col(A) is lost ( capturing r/m of gradient energy in expectation under Gaussian initialization and suffering zero update energy in orthogonal directions under truncated-SVD initialization).

supports Source: LoRA-FA: Memory-Efficient Low-Rank Adaptation for Large Language Models Fine-Tuning · Locator: Appendix E.3 (Initialization of A) & Appendix E.5 (Limitations), pp. 23-25

⚠️ Curated Limitations & Negative Results

Freezing A restricts all weight updates to the fixed r-dimensional column subspace col(A_0) throughout training, capturing only an r/m fraction of arbitrary gradient energy in expectation.
LoRA-FA eliminates the input activation cache of the adapter branch but does not remove non-adapter Transformer activations such as attention scores, LayerNorm, or GeLU intermediates unless paired with FlashAttention or selective activation checkpointing.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	LoRA-FA fine-tunes linear layers in deep artificial neural networks with reduced activation memory and closed-form gradient preconditioning.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r666-primary-pdf`	LoRA-FA: Memory-Efficient Low-Rank Adaptation for Large Language Models Fine-Tuning Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, Bo Li	primary paper full_text	ID: https://dblp.org/search/publ/api?q=LoRA-FA+Longteng+Zhang&format=xml sha256:92cbd65b03a4…
`r666-arxiv-abs`	LoRA-FA: Memory-Efficient Low-Rank Adaptation for Large Language Models Fine-Tuning (arXiv abstract) Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, Bo Li	primary paper abstract_only	ID: https://arxiv.org/abs/2308.03303 sha256:71875b5a677f…
`r666-metadata-cr`	LoRA-FA: Memory-Efficient Low-Rank Adaptation for Large Language Models Fine-Tuning (Crossref metadata) Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, Bo Li	primary paper abstract_only	ID: https://dblp.org/search/publ/api?q=LoRA-FA+Longteng+Zhang&format=xml sha256:7d58a9788804…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T12:29:00Z

Updated: 2026-10-19T12:29:00Z

Source File: data/curated/LoRAFA.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories