Home / Curated / Algorithms / AdaLomo (Low-Memory Optimization with Adaptive Learning Rate)

AdaLomo (Low-Memory Optimization with Adaptive Learning Rate)

Algorithms AIMech:AdaLomoLowMemoryAdaptiveLearningRateOptimization DRAFT

Aliases: AdaLomo, AdaLomo Optimizer, Low-Memory Optimization with Adaptive Learning Rate, Fused-Backward Factored Second-Moment Adaptive Optimizer

A memory-efficient full-parameter adaptive optimization algorithm for large language models that fuses gradient backpropagation with in-place parameter updates to maintain O(1) gradient memory while providing per-parameter adaptive learning rates via rank-1 non-negative matrix factorization of the second-order gradient moment and single-pass parameter-wise root-mean-square grouped update normalization.

Identity Notes & Concept Scope Distinct from LOMO (AIMech:LomoFusedBackwardLowMemoryOptimization), which applies naive non-adaptive stochastic gradient descent during the fused backward pass and requires two backward passes whenever global gradient normalization is used; AdaLomo incorporates per-parameter second-moment scaling via row/column moving-average factors (r_{t,i}, c_{t,i}) and normalizes each tensor's update in a single fused backward pass via grouped update normalization. Distinct from Adafactor (AIMech:AdafactorSublinearMemoryAdaptiveOptimization), which materializes full-model gradients before the optimizer step and therefore consumes roughly 2.5x the memory of AdaLomo.

Claims & Grounded Evidence

5 Claims

Mechanism #adalomo-fused-backward-nmf-second-moment-mechanism

During the backward pass, as soon as the gradient g_{t,i} of parameter tensor theta_{t-1,i} in R^{m x n} is computed, AdaLomo updates row and column second-moment vectors r_{t,i} = beta * r_{t-1,i} + (1 - beta) * g_{t,i}^2 * 1_n in R^{m x 1} and c_{t,i} = beta * c_{t-1,i} + (1 - beta) * 1_m^T * g_{t,i}^2 in R^{1 x n}, reconstructs the non-negative matrix factorization second-moment estimate v_{t,i} = (r_{t,i} * c_{t,i}) / (1_m^T * r_{t,i}), updates theta_{t,i} immediately, and discards g_{t,i-1}, reducing optimizer-state storage to m + n values per matrix and gradient storage to at most two consecutive parameter tensors.

supports Source: AdaLomo: Low-memory Optimization with Adaptive Learning Rate · Locator: Section 2.1, Section 2.2 Equations (5)-(7), and Section 3.1 Algorithm 1 lines 5-13, pages 2-4

Mechanism #adalomo-grouped-update-normalization-mechanism

To avoid the two backward passes required by global gradient norm clipping in fused-backward training, AdaLomo applies grouped update normalization independently to each parameter tensor's raw adaptive update u_{t,i} within a single backward pass via u_hat_{t,i} = (u_{t,i} / max(1, RMS(u_{t,i}))) * max(epsilon, RMS(theta_{t-1,i})), where RMS(u) = sqrt((1/n) * sum_{k=1}^n u_k^2), scaling the step size proportionally to the parameter tensor's root-mean-square magnitude.

supports Source: AdaLomo: Low-memory Optimization with Adaptive Learning Rate · Locator: Section 3.1 Algorithm 1 line 11 and Section 3.2 Grouped Update Normalization, pages 4-5

Scope #adalomo-second-moment-saddle-and-lipschitz-scope

Ablation analysis isolating Adam's first-order momentum (Equation 3) versus second-order variance scaling (Equation 4) demonstrates that second-order moment estimation is the primary factor enabling escape from suboptimal local minima and compensating for unbounded layer-wise Lipschitz variations in Transformer self-attention.

supports Source: AdaLomo: Low-memory Optimization with Adaptive Learning Rate · Locator: Section 2.2 Theoretical and Empirical Analysis, Equations (3)-(4), Figure 1, and Appendix A Figure 6, pages 3-4, 13

Application #adalomo-llm-instruction-tuning-and-pretraining-application

Applied to instruction-tuning LLaMA-7B/13B/30B/65B on Alpaca-GPT4 (evaluated on MMLU, BBH, GSM8K, HumanEval, and AlpacaFarm), domain continual pre-training on Chinese and Python corpora, and from-scratch pre-training of a 1.1B LLaMA model on C4, AdaLomo matches AdamW convergence and benchmark accuracy while reducing total memory footprint to the level of LoRA (~40% of Adafactor's memory).

supports Source: AdaLomo: Low-memory Optimization with Adaptive Learning Rate · Locator: Section 1 Table 1, Section 4.1-4.4 Table 2, Figures 2-5, and Appendix F Table 8, pages 2, 5-9, 17

Limitation #adalomo-activation-memory-and-early-warmup-limitation

Because AdaLomo reduces only gradient and optimizer-state memory and omits first-order momentum, its overall memory savings are limited in regimes dominated by activation memory, and it can exhibit minor early-warmup loss fluctuations on smaller models as well as a modest throughput reduction relative to non-adaptive LOMO.

supports Source: AdaLomo: Low-memory Optimization with Adaptive Learning Rate · Locator: Section 4.2 page 7, Section 4.4 page 8, and Limitations Section, page 9

⚠️ Curated Limitations & Negative Results

Reduces memory occupied by gradients and optimizer states but does not compress intermediate activation memory.
Omitting first-order momentum can cause slight loss fluctuations during the initial warmup phase compared to AdamW.
Incurs a small per-step computational overhead for factored second-moment updates and grouped RMS normalization relative to vanilla LOMO.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	AdaLomo is a memory-efficient full-parameter adaptive gradient optimizer for training and instruction-tuning deep neural network and Transformer language models.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r635-primary-pdf`	AdaLomo: Low-memory Optimization with Adaptive Learning Rate Kai Lv, Hang Yan, Qipeng Guo, Haijun Lv, Xipeng Qiu	primary paper full_text	ID: https://dblp.org/rec/conf/acl/Lv0GLQ24 sha256:65bd0e33ca1c…
`r635-arxiv-abs`	AdaLomo: Low-memory Optimization with Adaptive Learning Rate (arXiv abstract) Kai Lv, Hang Yan, Qipeng Guo, Haijun Lv, Xipeng Qiu	primary paper abstract_only	ID: https://arxiv.org/abs/2310.10195 sha256:65c13009e2c5…
`r635-metadata-cr`	AdaLomo: Low-memory Optimization with Adaptive Learning Rate (Crossref metadata) Kai Lv, Hang Yan, Qipeng Guo, Haijun Lv, Xipeng Qiu	primary paper abstract_only	ID: https://dblp.org/rec/conf/acl/Lv0GLQ24 sha256:603bbb15629f…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T11:58:00Z

Updated: 2026-10-19T11:58:00Z

Source File: data/curated/AdaLomoLowMemoryAdaptiveLearningRateOptimization.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories