Home / Curated / Algorithms / AdEMAMix (Adaptive EMA Mixture Optimizer)

AdEMAMix (Adaptive EMA Mixture Optimizer)

Algorithms AIMech:AdEMAMixAdaptiveEMAMixtureOptimization DRAFT

Aliases: AdEMAMix Optimizer, Adaptive Exponential Moving Average Mixture Optimizer, Dual-EMA AdamW (AdEMAMix)

An AdamW-based adaptive optimization algorithm (Algorithm 1) that replaces the single first-moment exponential moving average with a linear combination of a bias-corrected fast-changing EMA m_hat_1^(t) = m_1^(t) / (1 - beta_1^t) (typically beta_1 = 0.9, half-life t_half approx 6 steps) and an uncorrected slow-changing EMA m_2^(t) = beta_3^(t) m_2^(t-1) + (1 - beta_3^(t)) g^(t) (typically beta_3 = 0.9999, half-life t_half approx 6930 steps), updating parameters via theta^(t) = theta^(t-1) - eta^(t) [(m_hat_1^(t) + alpha^(t) m_2^(t)) / (sqrt(nu_hat^(t)) + epsilon) + lambda theta^(t-1)] with warmup schedulers f_alpha and f_{beta_3} that increase alpha^(t) and the EMA half-life t_half(beta_3^(t)) = ln(0.5)/ln(beta_3^(t)) - 1 linearly over T_{alpha,beta_3} steps.

Identity Notes & Concept Scope Distinct from standard AdamW in that a single EMA cannot simultaneously assign high weight to recent gradients and non-negligible weight to gradients thousands of steps old; AdEMAMix maintains two separate first-moment buffers m_1 and m_2 (or sets beta_1 = 0 so m_hat_1^(t) = g^(t) without extra memory) and schedules beta_3^(t) = exp(ln(beta_start) ln(beta_3) / ((1 - t/T) ln(beta_3) + (t/T) ln(beta_start))). Distinct from nested-EMA methods (DEMA, AdMeta) which attenuate rather than preserve very old gradients.

Claims & Grounded Evidence

3 Claims

Mechanism #c-r597-fast-slow-ema-mixture-and-linear-halflife-scheduler

Because an EMA with parameter beta has cumulative half-weight horizon t_half(beta) = ln(0.5)/ln(beta) - 1 (approx 6 steps for beta = 0.9 vs 6930 steps for beta = 0.9999), AdEMAMix combines a fast EMA m_hat_1^(t) with a slow EMA alpha^(t) m_2^(t) and schedules beta_3^(t) = f^{-1}((1 - mu) f(beta_start) + mu f(beta_3)) (Eq. f_{beta_3}, Appendix A.1) so that t_half increases linearly with mu = t / T_{beta_3}, preventing early training divergence while damped orthogonal updates accelerate progress along narrow nonconvex valleys (Figures 2, 13-14, 18, 27).

supports Source: The AdEMAMix Optimizer: Better, Faster, Older · Locator: Figure 2, p. 3; Section 3, Equations (AdEMAMix), (f_alpha), (f_{beta_3}) & Algorithm 1, pp. 4-6; Appendix A.1, p. 18; Appendix C.1.4-C.1.9, pp. 29-37

Application #c-r597-llm-token-efficiency-and-reduced-batch-forgetting

On RedPajama Transformer language models (110M, 330M, 1.3B parameters), Mamba (168M on FineWeb), and ImageNet-21k ViTs (24M, 86M), AdEMAMix consistently outperforms tuned AdamW — matching a 1.3B AdamW model trained on 197B tokens using only 101B tokens (a 49% token reduction) — and exhibits significantly slower forgetting of held-out injected training batches over tens of thousands of steps (Figures 1, 3-4, 6, 9-10).

supports Source: The AdEMAMix Optimizer: Better, Faster, Older · Locator: Figure 1, p. 2; Figures 3-6 & Section 4.1-4.3, pp. 5-10; Appendix C.1.1-C.1.2, Table 7 & Figures 8-10, pp. 25-27

Limitation #c-r597-memory-overhead-short-horizon-and-distribution-shift-limitations

When beta_1 != 0, AdEMAMix allocates a third parameter-sized optimizer buffer m_2 (33% more optimizer state memory than AdamW unless beta_1 = 0 is used at the cost of noisier loss spikes), and retaining very old gradients is less effective in short-horizon or high-epoch overfitting regimes (such as 320-epoch ImageNet-1k ViT) or under sudden distribution shifts.

supports Source: The AdEMAMix Optimizer: Better, Faster, Older · Locator: Section 3, Computational overheads & Limitations, pp. 6-7; Section 4.3, Figure 6c, p. 10; Appendix C.1.5 & C.1.7, pp. 31, 34

⚠️ Curated Limitations & Negative Results

Requires an additional parameter-sized momentum buffer m_2 when beta_1 != 0, increasing optimizer state memory over AdamW unless beta_1 is set to 0.
Retaining gradient information over thousands of steps can hinder rapid adaptation under sudden distribution shifts or provide limited benefit in short-iteration, multi-epoch overfitting regimes.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	AdEMAMix is a dual-EMA adaptive optimizer designed for training Transformer, Mamba, and Vision Transformer artificial neural networks.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r597-primary-pdf`	The AdEMAMix Optimizer: Better, Faster, Older Matteo Pagliardini, Pierre Ablin, David Grangier	primary paper full_text	ID: https://dblp.org/rec/conf/iclr/PagliardiniAG25 sha256:f3502dffb875…
`r597-arxiv-abs`	The AdEMAMix Optimizer: Better, Faster, Older (arXiv abstract) Matteo Pagliardini, Pierre Ablin, David Grangier	primary paper abstract_only	ID: https://arxiv.org/abs/2409.03137 sha256:6d191babee12…
`r597-metadata-cr`	The AdEMAMix Optimizer: Better, Faster, Older (Crossref metadata) Matteo Pagliardini, Pierre Ablin, David Grangier	primary paper abstract_only	ID: https://dblp.org/rec/conf/iclr/PagliardiniAG25 sha256:f4d5fa655e42…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T11:20:00Z

Updated: 2026-10-19T11:20:00Z

Source File: data/curated/AdEMAMixAdaptiveEMAMixtureOptimization.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories