Home / Curated / Algorithms / Weight-Decomposed Low-Rank Adaptation (DoRA)

Weight-Decomposed Low-Rank Adaptation (DoRA)

Algorithms AIMech:DoraWeightDecomposedLowRankAdaptation DRAFT

Aliases: DoRA, Weight-Decomposed Low-Rank Adaptation, DoRA PEFT, Magnitude-Direction Decomposed Low-Rank Adaptation

Weight-Decomposed Low-Rank Adaptation (DoRA) is a parameter-efficient fine-tuning (PEFT) method that decomposes a pretrained weight matrix W_0 in R^{d x k} into a trainable 1 x k column-wise magnitude vector m = ||W_0||_c and a directional matrix V = W_0, and applies Low-Rank Adaptation (LoRA) exclusively to the directional component: W' = m * (V + Delta V) / ||V + Delta V||_c = m * (W_0 + B A) / ||W_0 + B A||_c, where B in R^{d x r} and A in R^{r x k} with rank r << min(d, k) while W_0 remains frozen. By decoupling magnitude and direction updates, DoRA reproduces the negative-slope magnitude-direction update correlation characteristic of full fine-tuning (FT) while merging W' back into a single d x k matrix prior to deployment with zero additional inference latency.

Identity Notes & Concept Scope Distinct from standard Low-Rank Adaptation (LoRA, Hu et al., 2022), which updates W' = W_0 + B A directly without column-wise directional normalization ||W_0 + B A||_c or an independent magnitude vector m, causing LoRA to exhibit a positive coupling between magnitude and directional changes that struggles to make fine-grained directional adjustments without simultaneous large magnitude shifts. Also distinct from Weight Normalization (Salimans & Kingma, 2016), which trains full weight matrices V from random initialization rather than freezing W_0 and adapting V via a low-rank increment Delta V = B A.

Claims & Grounded Evidence

4 Claims

Mechanism #r651-claim-magnitude-direction-decomposition

DoRA initializes the trainable magnitude vector m = ||W_0||_c in R^{1 x k} and frozen directional base V = W_0 so that W' = m * (W_0 + B A) / ||W_0 + B A||_c exactly equals W_0 at initialization (when B = 0), and during fine-tuning updates only m (k parameters) and the low-rank factors B and A, adding just k / (d * k) = 1/d fractional parameter overhead over LoRA.

supports Source: DoRA: Weight-Decomposed Low-Rank Adaptation · Locator: Section 3.2, Equation (5) and Figure 2, p. 4-5

Mechanism #r651-claim-gradient-stabilization-and-detached-norm

Under DoRA's reparameterization V' = V + Delta V, the gradients with respect to magnitude m and directional increment Delta V are scaled by 1 / ||V'||_c, and detaching ||V + Delta V||_c from backpropagation (treating ||V'||_c as a constant C during the backward pass) leaves the magnitude gradient nabla_m L unchanged while eliminating computational graph overhead for the column norm.

supports Source: DoRA: Weight-Decomposed Low-Rank Adaptation · Locator: Section 3.3 (Gradient Analysis of DoRA), Equations (6)-(9) and Table 2, p. 5-6

Application #r651-claim-llm-vlm-finetuning-applications

DoRA consistently outperforms LoRA across commonsense reasoning fine-tuning on LLaMA-7B/13B, LLaMA2-7B, and LLaMA3-8B, visual instruction tuning on LLaVA-1.5-7B, and image/video-text understanding on VL-BART, and retains its accuracy advantage even at very low ranks (r = 4 to 8) while merging into W_0 with zero inference overhead.

supports Source: DoRA: Weight-Decomposed Low-Rank Adaptation · Locator: Section 4.1-4.3, Tables 3, 4, 5 & Figure 4, p. 6-9

Limitation #r651-claim-training-memory-and-norm-compute-limitation

Although detaching ||V + Delta V||_c substantially reduces backward-pass memory compared to full backpropagation through the column norm, DoRA still requires materializing or computing the column-wise norms ||W_0 + B A||_c during each forward pass of training, incurring a modest training-time memory and compute overhead relative to vanilla LoRA.

supports Source: DoRA: Weight-Decomposed Low-Rank Adaptation · Locator: Section 3.3, Table 2 (Training memory savings of detaching ||V + Delta V||_c), p. 5-6

⚠️ Curated Limitations & Negative Results

Computing the column-wise Euclidean norm ||W_0 + B A||_c during every training step adds a modest forward-pass compute and activation-memory overhead compared to standard LoRA.
When combined with quantized base weights (e.g., QDoRA / 4-bit NF4), computing ||W_0 + B A||_c requires dequantizing W_0 or maintaining precomputed norm approximations.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	DoRA is a parameter-efficient fine-tuning algorithm for adapting linear projection layers in pretrained deep neural networks and large language models.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r651-primary-pdf`	DoRA: Weight-Decomposed Low-Rank Adaptation Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, KT Cheng, Min-Hung Chen	primary paper full_text	ID: https://dblp.org/rec/conf/icml/LiuWY0WCC24.html sha256:f048986e4d0b…
`r651-arxiv-abs`	DoRA: Weight-Decomposed Low-Rank Adaptation (arXiv abstract) Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, KT Cheng, Min-Hung Chen	primary paper abstract_only	ID: https://arxiv.org/abs/2402.09353 sha256:6e7d6de337ee…
`r651-metadata-cr`	DoRA: Weight-Decomposed Low-Rank Adaptation (Crossref metadata) Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, KT Cheng, Min-Hung Chen	primary paper abstract_only	ID: https://dblp.org/rec/conf/icml/LiuWY0WCC24.html sha256:851fa6217aa4…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T12:14:00Z

Updated: 2026-10-19T12:14:00Z

Source File: data/curated/DoraWeightDecomposedLowRankAdaptation.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories