Home / Curated / Algorithms / ReLoRA

ReLoRA

Algorithms AIMech:ReLoRA DRAFT

Aliases: Restarted Low-Rank Adaptation (ReLoRA), ReLoRA High-Rank Training Through Low-Rank Updates, Warm-Started Periodic Reset Low-Rank Pre-Training

A parameter-efficient high-rank neural network training algorithm that transitions from an initial full-rank warm-start phase to sequential low-rank adaptation updates Delta W = sum_{k=1}^N s W_A^k W_B^k by periodically merging learned low-rank factors into the frozen base weights, reinitializing the low-rank matrices, partially resetting Adam optimizer states via magnitude pruning, and applying a jagged cosine learning rate schedule with rapid zero-learning-rate re-warmups.

Identity Notes & Concept Scope Distinct from standard LoRA (AIMech:LoRA, which trains a single pair of low-rank matrices whose update rank is bounded above by r without restarts or warm-start pre-training), from Chain of LoRA (AIMech:ChainOfLoRA, which fine-tunes already pretrained models via a Frank-Wolfe conditional-gradient chain with step-down rank schedules rather than pre-training transformers with a full-rank warm start, magnitude-pruned partial Adam moment resets, and a jagged cosine learning rate schedule), and from GaLore (AIMech:GaLoreGradientLowRankProjection, which projects full-parameter gradients into low-rank subspaces via SVD rather than parameterizing linear layers with merged-and-reinitialized LoRA adapters). Formulated by Lialin et al. (ICLR 2024, arXiv:2307.05695).

Claims & Grounded Evidence

4 Claims

Mechanism #relora-sum-of-ranks-periodic-merge-and-reinit

ReLoRA exploits the subadditivity of matrix rank, rank(A + B) <= rank(A) + rank(B), by wrapping frozen linear layers W^i with rank-r factors (W_A^i, W_B^i) and every q steps merging the update W^i <- W^i + s W_A^i W_B^i into the base weights and reinitializing W_A^i with Kaiming initialization and W_B^i with zeros, accumulating a high-rank total update Delta W = sum_{k=1}^N s W_A^k W_B^k whose singular value spectrum closely matches full-rank training.

supports Source: ReLoRA: High-Rank Training Through Low-Rank Updates · Locator: Section 2 Method, Eq. 1-2, Algorithm 1, and Section 4 Figures 3-4, p. 2-6

Mechanism #relora-partial-optimizer-pruning-and-jagged-cosine-schedule

To prevent historical Adam first and second moments (M_{W_A}, V_{W_A}) with high beta_1, beta_2 from constraining reinitialized factors W_A^{k+1} to the previous subspace while avoiding divergence from a full reset, ReLoRA prunes 90% to 99% of the lowest-magnitude optimizer state entries at each restart and pairs the reset with a jagged cosine learning rate schedule that drops the learning rate to zero and warms it back up over 50 to 100 steps.

supports Source: ReLoRA: High-Rank Training Through Low-Rank Updates · Locator: Section 2 Method, Figure 2, Algorithm 1 Lines 17-20, and Section 4.2 Table 6, p. 2-4, 8

Application #relora-transformer-pretraining-application

ReLoRA is applied to pre-training transformer language models from 60M up to 1.3B parameters on the C4 corpus following an initial full-rank warm-start phase, achieving perplexity comparable to full-rank training while saving up to 5.5 GB of GPU RAM per device and improving training throughput by 9% to 40% across hardware setups.

supports Source: ReLoRA: High-Rank Training Through Low-Rank Updates · Locator: Section 3 Experiments & Section 4 Results, Tables 2, 4, and 7, p. 4-8

Limitation #relora-full-rank-warmstart-requirement-and-frequent-reset-degradation

Training from random initialization with ReLoRA requires a substantial initial full-rank warm-start phase (typically 25% to 33% of total update steps) to avoid significant perplexity degradation, and overly frequent online LoRA merges (e.g., every 100 steps) or restarts without the jagged learning rate schedule degrade performance or cause training divergence.

supports Source: ReLoRA: High-Rank Training Through Low-Rank Updates · Locator: Section 4.1 Negative results: Online ReLoRA, Table 5, and Section 4.2 Table 6, p. 7-8

⚠️ Curated Limitations & Negative Results

Requires an initial full-rank training warm-start phase when training from scratch before switching to low-rank updates.
Overly frequent online merges (Online ReLoRA) or optimizer resets without jagged zero-LR re-warmup degrade perplexity or cause divergence, and ReLoRA does not outperform standard single-stage LoRA on downstream GLUE fine-tuning.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	ReLoRA trains high-rank transformer neural networks through sequential merged low-rank updates.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r682-primary-pdf`	ReLoRA: High-Rank Training Through Low-Rank Updates Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, Anna Rumshisky	primary paper full_text	ID: https://dblp.org/rec/conf/iclr/LialinMSR24.xml sha256:456aa1b49414…
`r682-arxiv-abs`	ReLoRA: High-Rank Training Through Low-Rank Updates (arXiv abstract) Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, Anna Rumshisky	primary paper abstract_only	ID: https://arxiv.org/abs/2307.05695 sha256:97416bde109d…
`r682-metadata-cr`	ReLoRA: High-Rank Training Through Low-Rank Updates (Crossref metadata) Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, Anna Rumshisky	primary paper abstract_only	ID: https://dblp.org/rec/conf/iclr/LialinMSR24.xml sha256:da95b6d8065d…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T12:45:00Z

Updated: 2026-10-19T12:45:00Z

Source File: data/curated/ReLoRA.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories