Adan (Adaptive Nesterov Momentum Algorithm)
Algorithms AIMech:AdanAdaptiveNesterovMomentumAlgorithm DRAFT
Aliases: Adan Optimizer, Adaptive Nesterov Momentum Estimation, NME Adaptive Nesterov Optimizer
An adaptive first-order stochastic optimization algorithm for deep neural networks that reformulates Nesterov accelerated gradient via Nesterov Momentum Estimation (NME) so that lookahead extrapolation requires zero additional gradient oracle evaluations. At step k, Adan updates a first-order gradient EMA m_k = (1 - beta_1) m_{k-1} + beta_1 g_k, a first-order gradient-difference EMA v_k = (1 - beta_2) v_{k-1} + beta_2 (g_k - g_{k-1}), a second-order composite moment n_k = (1 - beta_3) n_{k-1} + beta_3 [g_k + (1 - beta_2)(g_k - g_{k-1})]^2, an elementwise adaptive step size eta_k = eta / (sqrt(n_k) + eps), and a proximal weight-decay iterate update theta_{k+1} = (1 + lambda_k eta)^{-1} (theta_k - eta_k circ (m_k + (1 - beta_2) v_k)).
Identity Notes & Concept Scope
Distinct from Nadam (Dozat 2016), AdamW (Loshchilov and Hutter 2019), Lion (AIMech:LionEvoSignMomentumOptimizer), and AdEMAMix (AIMech:AdEMAMixDualExponentialMovingAverageOptimizer): Adan derives its update from an algebraic rearrangement of Nesterov's extrapolated gradient (Lemma 1, Eq. 4) that maintains a dedicated gradient-difference EMA v_k with its own decay coefficient beta_2, preconditioning the combined vector m_k + (1 - beta_2) v_k by the RMS of the composite Nesterov perturbation g_k + (1 - beta_2)(g_k - g_{k-1}) and applying exact proximal scaling (1 + lambda_k eta)^{-1} for decoupled L2 regularization.
Claims & Grounded Evidence
5 Claims
Mechanism
#nesterov-momentum-estimation-lemma-1-equivalence
Lemma 1 (Section 2.1, p. 3) proves that the lookahead Nesterov accelerated gradient recursion theta'_k = theta_k - eta (1 - beta_1) m_{k-1}, m_k = (1 - beta_1) m_{k-1} + g'_k, theta_{k+1} = theta_k - eta m_k is algebraically identical to defining u_k := theta'_k with update u_{k+1} = u_k - eta m'_k, where m'_k = (1 - beta_1) m'_{k-1} + [g'_k + (1 - beta_1)(g'_k - g'_{k-1})] = sum_{t=0}^k (1 - beta_1)^t [g'_{k-t} + (1 - beta_1)(g'_{k-t} - g'_{k-t-1})], eliminating the need to compute gradients at a separate extrapolated point.
Mechanism
#proximal-weight-decay-fixed-point-and-separability
Adan derives its weight-decay step theta_{k+1} = (1 + lambda_k eta)^{-1} (theta_k - eta_k circ (m_k + (1 - beta_2) v_k)) as the exact first-order approximation to the scaled proximal operator argmin_theta { (lambda_k eta / 2) ||theta||_2^2 + (1/2) ||theta - (theta_k - eta_k circ u_k)||_{1/sqrt(eta_k)}^2 } (Eq. 5, p. 4), and Proposition 3 (p. 5) proves that every fixed point theta* of Adan satisfies nabla F(theta*) + lambda (c_l + eps) theta* = 0, dynamically decoupling the optimization objective from weight-decay scaling.
Scope
#smooth-and-holder-nonconvex-stochastic-complexity-theorems-1-and-2
Under L-smooth nonconvex stochastic objectives with coordinate-wise bounded gradient norms ||g_k||_infty <= c_infty and variance E||g_k - nabla F(theta_k)||_2^2 <= sigma^2 (Assumptions 1-3), Theorem 1 (Section 2.3, p. 5) proves that Adan achieves an epsilon-approximate first-order stationary point in O(c_infty sigma^2 epsilon^{-4}) stochastic gradient evaluations, matching the lower bound under weakened coordinate-wise boundedness, while Theorem 2 (p. 6) establishes convergence under Holder-continuous gradients with parameter nu in (0, 1) without global Lipschitz smoothness.
Application
#vision-nlp-and-rl-training-acceleration
Across ViT, Swin, ConvNeXt, ResNet, MAE, GPT-2, Transformer-XL, BERT, and MuJoCo PPO benchmarks, Adan matches or exceeds the test accuracy and reward of Adam, AdamW, LAMB, and AdaBelief while using half the training epochs on ImageNet vision backbones.
Limitation
#extra-gradient-difference-state-and-three-beta-hyperparameters
Adan stores three running state tensors per parameter (m_k, v_k, n_k) plus the previous step gradient g_{k-1}—increasing optimizer state memory relative to two-state AdamW—and introduces three momentum decay hyperparameters (beta_1, beta_2, beta_3) that require coordinated selection.
⚠️ Curated Limitations & Negative Results
- Maintains an additional first-order gradient-difference state tensor v_k and previous gradient g_{k-1} beyond standard AdamW's two-moment state.
- Introduces three momentum decay coefficients (beta_1, beta_2, beta_3) whose finite-time stability depends on beta_2 controlling the weight of high-variance stochastic gradient differences g_k - g_{k-1}.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ArtificialNeuralNetwork
Artificial Neural Network
|
related_match | Adan is an adaptive Nesterov momentum optimization algorithm for training artificial neural networks across vision, language, and reinforcement learning tasks. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
r602-primary-pdf
|
Xingyu Xie, Pan Zhou, Huan Li, Zhouchen Lin, Shuicheng Yan
|
primary paper
full_text
|
ID: https://dblp.org/rec/journals/pami/XieZLLY24
sha256:789dd2d5084f…
|
r602-arxiv-abs
|
Xingyu Xie, Pan Zhou, Huan Li, Zhouchen Lin, Shuicheng Yan
|
primary paper
abstract_only
|
ID: https://arxiv.org/abs/2208.06677
sha256:0fc34443464f…
|
r602-metadata-cr
|
Xingyu Xie, Pan Zhou, Huan Li, Zhouchen Lin, Shuicheng Yan
|
primary paper
abstract_only
|
ID: https://dblp.org/rec/journals/pami/XieZLLY24
sha256:fed69d316aeb…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T11:25:00Z
Updated: 2026-10-19T11:25:00Z
Source File:
data/curated/AdanAdaptiveNesterovMomentumAlgorithm.yaml
· Schema:
curated_concept.yaml
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories