Home / Curated / Algorithms / rsLoRA

rsLoRA

Algorithms AIMech:RsLoRA DRAFT

Aliases: Rank-Stabilized LoRA, Rank-Stabilized Low-Rank Adaptation, Rank Stabilization Scaling Factor for LoRA

A rank-stabilized low-rank adaptation algorithm that scales the low-rank adapter product BA by gamma_r = alpha / sqrt(r) instead of the conventional alpha / r factor, ensuring that adapter activation moments and backpropagated input gradients remain Theta_r(1) as the rank r increases.

Identity Notes & Concept Scope Distinct from AIMech:LowRankAdaptation (LoRA), which scales the adapter product B A by gamma_r = alpha / r and consequently suffers an O(1/r) collapse in effective update magnitude and gradient norms as r grows, and from AIMech:LoraPlusEfficientLowRankAdaptationWithAsymmetricLearningRates (LoRA+), which addresses the width-d asymmetry between A and B via unequal learning rates eta_B = lambda eta_A. rsLoRA (Damjan Kalajdzievski, arXiv:2312.03732) analyzes the infinite-rank limit r -> infinity of an adapter x_{out} = (W + gamma_r B A) x_{in} + b initialized with B_0 = 0 and A_{i,j} ~ N(0, sigma_A^2) independent of r. Theorem 3.2 proves that after n gradient steps, E_{A_0}[gamma_r B_n A_n] = -gamma_r^2 r sigma_A eta sum_{k=0}^{n-1} v_k x_k^T + O_r(gamma_r^3 r), so both the m-th forward activation moments and the backpropagated input gradients are Theta_r((gamma_r^2 r)^m), which is non-collapsing and stable (Theta_r(1)) if and only if gamma_r in Theta_r(1 / sqrt(r)).

Claims & Grounded Evidence

4 Claims

Mechanism #rslora-inverse-sqrt-rank-scaling-factor-theorem

For an adapter gamma_r B A with B_0 = 0 and A_0 entries drawn i.i.d. with mean 0 and r-independent variance sigma_A, the expected update after n steps scales as E_{A_0}[gamma_r B_n A_n] in Theta_r(gamma_r^2 r), so setting gamma_r = alpha / sqrt(r) is the unique scaling exponent (up to a constant) that keeps forward output moments and backward input gradients at Theta_r(1) as r -> infinity.

supports Source: A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA · Locator: Section 3 (Definition 3.1 & Theorem 3.2) & Appendix A (Proof of Theorem 3.2, Equations 6-11), pp. 3, 8-9

Mechanism #rslora-elimination-of-high-rank-gradient-collapse

Whereas standard LoRA scaling gamma_r = alpha / r causes parameter gradient norms to shrink by orders of magnitude as r increases from 4 to 2048—stunting learning so high ranks perform no better than rank 4—rsLoRA preserves consistent initial gradient norms across all ranks and unlocks monotonic perplexity improvements with increasing rank.

supports Source: A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA · Locator: Section 4 (Experimental Results, Figures 2-3) & Appendix B.1-B.4, pp. 4-5, 9-11

Application #rslora-application-llama2-and-gptj-instruction-and-math-finetuning

Applied to LLaMA-2-7B on OpenOrca with AdamW and SGD and to GPT-J-6B on GSM8K with Adafactor across ranks r in {4, 8, 32, 128, 512, 2048}, rsLoRA enables smooth training-compute-versus-perplexity scaling while merging (W + (alpha/sqrt(r)) B A) into the base weights with zero inference overhead.

supports Source: A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA · Locator: Section 4 & Appendix B.2 (Figures 2, 4, 5), pp. 4-5, 9-10

Limitation #rslora-limitation-initialization-only-scaling-instability

Scaling only the initialization variance of A_0 by 1 / sqrt(r) without applying the multiplicative factor gamma_r = alpha / sqrt(r) to the adapter forward pass stabilizes gradients only at step 0 and becomes unstable later in training at large ranks (e.g., r = 2048), while higher ranks still incur proportional training memory and FLOPs.

supports Source: A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA · Locator: Section 4 & Appendix B.3 (Scaling at Initialization Only, Figure 6), pp. 5, 10

⚠️ Curated Limitations & Negative Results

Although rsLoRA merges into W with zero inference overhead, training at high ranks (e.g., r = 512 or 2048) still increases training-time adapter parameters, optimizer states, and FLOPs linearly with r.
Rank-correcting only the initialization distribution of A without keeping the multiplicative 1/sqrt(r) forward scaling factor fails to prevent training instability at large ranks.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	rsLoRA stabilizes low-rank adapter fine-tuning across arbitrary ranks in deep artificial neural networks.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r668-primary-pdf`	A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA Damjan Kalajdzievski	primary paper full_text	ID: https://dblp.org/search/publ/api?q=Rank+Stabilization+Scaling+Factor+LoRA+Kalajdzievski&format=xml sha256:8d45b0bc98bf…
`r668-arxiv-abs`	A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA (arXiv abstract) Damjan Kalajdzievski	primary paper abstract_only	ID: https://arxiv.org/abs/2312.03732 sha256:efa582cb19c3…
`r668-metadata-cr`	A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA (Crossref metadata) Damjan Kalajdzievski	primary paper abstract_only	ID: https://dblp.org/search/publ/api?q=Rank+Stabilization+Scaling+Factor+LoRA+Kalajdzievski&format=xml sha256:c17288a98cca…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T12:31:00Z

Updated: 2026-10-19T12:31:00Z

Source File: data/curated/RsLoRA.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories