Home / Curated / Algorithms / AdaMod

AdaMod

Algorithms AIMech:AdaMod DRAFT

Aliases: Adaptive and Momental Bound Method for Stochastic Learning, Momental Step-Size Upper-Bounded Adaptive Optimization, Exponentially Smoothed Learning-Rate Clipped Adam

An adaptive stochastic gradient optimization algorithm that restricts extreme coordinate-wise adaptive learning rates eta_t = alpha_t / (sqrt(v_hat_t) + epsilon) by maintaining an uncorrected exponential moving average s_t = beta_3 s_{t-1} + (1 - beta_3) eta_t (with s_0 = 0 and smoothing hyperparameter beta_3 in (0, 1)) of historical adaptive step sizes and clipping each coordinate's effective step size from above via eta_hat_t = min(eta_t, s_t) before updating parameters x_t = x_{t-1} - eta_hat_t odot m_hat_t.

Identity Notes & Concept Scope Distinct from AdaBound/AMSBound (AIMech:AdaBoundAndAMSBound, which clips step sizes to a deterministic scalar schedule [eta_l(t), eta_u(t)] converging to vanilla SGD) and from RAdam (AIMech:RectifiedAdam, which uses an analytical tractable-variance switch at rho_t > 4). AdaMod preserves per-coordinate adaptive step-size heterogeneity throughout training while using the uncorrected EMA s_t = (1 - beta_3) sum_{k=0}^{t-1} beta_3^k eta_{t-k} both as an automatic early-stage warmup ramp and as a dynamic momental upper bound against anomalous gradient-induced learning-rate spikes.

Claims & Grounded Evidence

4 Claims

Mechanism #adamod-momental-step-size-upper-bound-mechanism

At iteration t, AdaMod computes the bias-corrected first and second moments m_hat_t = m_t / (1 - beta_1^t) and v_hat_t = v_t / (1 - beta_2^t), forms the raw adaptive learning rate vector eta_t = alpha_t / (sqrt(v_hat_t) + epsilon), updates the uncorrected exponential moving average s_t = beta_3 s_{t-1} + (1 - beta_3) eta_t = (1 - beta_3) sum_{k=0}^{t-1} beta_3^k eta_{t-k} with s_0 = 0, caps the step size via eta_hat_t = min(eta_t, s_t), and updates parameters via x_t = x_{t-1} - eta_hat_t odot m_hat_t, reducing exactly to Adam when beta_3 = 0.

supports Source: An Adaptive and Momental Bound Method for Stochastic Learning · Locator: Methods Section, Algorithm 2 and Equations 1-3, p. 3
supports Source: An Adaptive and Momental Bound Method for Stochastic Learning (Crossref metadata) · Locator: Bibliographic metadata for Ding et al. 2019

Scope #adamod-intrinsic-warmup-and-spike-suppression-scope

Because s_0 = 0 is intentionally left without bias correction, s_t starts at (1 - beta_3) eta_1 (reducing initial step-size spikes above 10^4 by a factor of 1 - beta_3 in {10^{-3}, 10^{-4}} for beta_3 in {0.999, 0.9999}) and acts as a long-term memory filter over approximately 1 / (1 - beta_3) steps that prevents sudden large learning-rate spikes when v_hat_t is transiently small.

supports Source: An Adaptive and Momental Bound Method for Stochastic Learning · Locator: Introduction (Figure 1) and Empirical Analysis on Learning Rates (Figures 2-3), p. 1-5

Application #adamod-vision-lm-and-unwarmed-transformer-applications

AdaMod improves test accuracy and perplexity over Adam, AMSGrad, and AdaBound on CIFAR-10 and CIFAR-100 (ResNet-34 and DenseNet-121), ImageNet (ResNet-18), Penn Treebank LSTM language modeling (1-, 2-, and 3-layer LSTMs), and WMT14 English-to-German and IWSLT14 German-to-English Transformer neural machine translation without requiring a manual learning-rate warmup schedule.

supports Source: An Adaptive and Momental Bound Method for Stochastic Learning · Locator: Experiments Section, Figures 4-8 and Tables 1-3, p. 4-9

Limitation #adamod-extra-state-vector-and-beta3-memory-horizon-limitation

AdaMod requires maintaining a third per-parameter state vector s_t alongside m_t and v_t (increasing optimizer state memory from 2P to 3P floats) and introduces the memory-horizon hyperparameter beta_3 whose effective window length 1 / (1 - beta_3) must align with the training horizon.

supports Source: An Adaptive and Momental Bound Method for Stochastic Learning · Locator: Methods Section (Algorithm 2) and Experiments Section (Table 3), p. 3, 8-9

⚠️ Curated Limitations & Negative Results

Stores an additional per-parameter step-size moving average vector s_t alongside m_t and v_t, increasing optimizer state memory from 2P to 3P floats.
Introduces an additional smoothing hyperparameter beta_3 in {0.999, 0.9999} whose effective memory horizon 1 / (1 - beta_3) can overly damp step sizes on very short training runs.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	AdaMod stabilizes adaptive optimization of deep convolutional, recurrent, and Transformer neural networks without manual warmup schedules.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r613-primary-pdf`	An Adaptive and Momental Bound Method for Stochastic Learning Jianbang Ding, Xuancheng Ren, Ruixuan Luo, Xu Sun	primary paper full_text	ID: https://dblp.org/rec/journals/corr/abs-1910-12249.xml sha256:086818082905…
`r613-arxiv-abs`	An Adaptive and Momental Bound Method for Stochastic Learning (arXiv abstract) Jianbang Ding, Xuancheng Ren, Ruixuan Luo, Xu Sun	primary paper abstract_only	ID: https://arxiv.org/abs/1910.12249 sha256:f47aad693c26…
`r613-metadata-cr`	An Adaptive and Momental Bound Method for Stochastic Learning (Crossref metadata) Jianbang Ding, Xuancheng Ren, Ruixuan Luo, Xu Sun	primary paper abstract_only	ID: https://dblp.org/rec/journals/corr/abs-1910-12249.xml sha256:5a2142bbe649…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T11:36:00Z

Updated: 2026-10-19T11:36:00Z

Source File: data/curated/AdaMod.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories