Aggregated Momentum (AggMo Passive Damping Optimizer)
Algorithms AIMech:AggMoAggregatedMomentumPassiveDampingOptimization DRAFT
Aliases: AggMo, Aggregated Momentum Optimizer, Passive Damping Aggregated Momentum and Beta-Averaged Momentum
A multi-velocity first-order momentum optimization method inspired by passive structural damping that maintains K parallel velocity vectors v_t^(1), ..., v_t^(K) associated with a damping vector beta = [beta^(1), ..., beta^(K)] (by default exponentially spaced as beta^(i) = 1 - a^{i-1} with scale factor a = 0.1 for i = 1, ..., K). At each step t, AggMo updates every velocity independently via v_t^(i) = beta^(i) v_{t-1}^(i) - nabla_theta f(theta_{t-1}) and updates the parameters by averaging the velocities: theta_t = theta_{t-1} + (gamma_t / K) sum_{i=1}^K v_t^(i) (or with per-velocity stepsizes gamma_t^(i) in Eq. 4, or continuous Beta(alpha, beta) density integration in Beta-Averaged Momentum, Eqs. 14-17).
Identity Notes & Concept Scope
Distinct from Classical Polyak Momentum, Nesterov Accelerated Gradient, and AdEMAMix (AIMech:AdEMAMixAdaptiveEMAMixtureOptimization): AggMo is a non-adaptive multi-frequency passive-damping momentum scheme whose dynamics evolve according to a (K + 1)-th order finite difference equation (Appendix C.1, Eq. 12) rather than a 2nd-order ODE discretization, and whose generalized K = 2 form with beta = [0, beta] and gamma_t^(1) = 2 gamma, gamma_t^(2) = 2 beta gamma exactly recovers Nesterov's half-step reparameterized iterate phi_t = theta_t + gamma beta v_t (Section 4, Eqs. 4-7).
Claims & Grounded Evidence
5 Claims
Mechanism
#nesterov-accelerated-gradient-recovery-as-k2-aggmo-special-case
Section 4 (Eqs. 4-7, p. 4) proves that generalized AggMo with K = 2 damping coefficients beta = [0, beta] and velocity learning rates gamma_t^(1) = 2 gamma, gamma_t^(2) = 2 beta gamma yields the update theta_t = theta_{t-1} + gamma beta^2 v_{t-1} - (1 + beta) gamma nabla_theta f(theta_{t-1}), which is identical at every iteration to Nesterov's accelerated gradient under the lookahead reparameterization phi_t = theta_t + gamma beta v_t.
Mechanism
#block-companion-dynamical-system-and-k-plus-1-order-finite-difference
On quadratic objectives f(x) = (1/2) x^T A x + b^T x with eigenvalues {lambda_j}, AggMo evolves as a (K + 1)-dimensional linear dynamical system per eigenmode with block matrix B_j (Appendix B & C.1, p. 12-13, 17) and satisfies a (K + 1)-th order finite difference equation (Eqs. 9-12, p. 18), interpolating smoothly between the critical-damping convergence rates of its constituent beta^(i) modes and achieving ~3x faster convergence than Nesterov momentum in the under-damped regime (Section 5.1, Figure 3, p. 5).
Scope
#online-convex-programming-regret-bound-and-beta-averaged-momentum
Under online convex programming with bounded gradients ||nabla f_t(theta)||_infty < G_infty, bounded domain diameter D, D_infty, stepsize gamma_t = gamma / sqrt(t), and decaying damping beta_t^(i) = beta^(i) lambda^t, Lemma 1 and Theorem 1 (Section 5.2, p. 5-6; Appendix C, p. 14-17) prove R(T) <= D_infty^2 sqrt(T) / gamma + (gamma sqrt(1 + log T) / (2K)) sum_{j=1}^d ||g_{1:T,j}||_4^2 sum_{i=1}^K (1 + beta^(i)) / (1 - beta^(i))^2 + (D^2 / (2K gamma (1 - lambda)^2)) sum_{i=1}^K beta^(i) = O(sqrt(T)); Appendix F (Eqs. 14-17, p. 21) further derives closed-form Beta-Averaged Momentum updates with weight E[b^k] = prod_{r=0}^{k-1} (alpha + r) / (alpha + beta + r).
Application
#autoencoder-cnn-resnet-without-batchnorm-and-lstm-benchmarks
Across MNIST deep autoencoders, CIFAR-10 CNN-5, CIFAR-10/100 ResNet-32 (including training ResNet-32 without batch normalization at large learning rates), and Penn Treebank 3-layer LSTM language modeling, AggMo achieves faster training convergence and lower or competitive validation/test loss compared to classical momentum, Nesterov momentum, and Adam while remaining stable at damping coefficients up to 0.999.
Limitation
#k-fold-velocity-memory-overhead-and-open-closed-form-spectral-analysis
AggMo stores K separate velocity vectors of the same dimension as the model parameters (increasing optimizer state memory K-fold over classical momentum), lacks a closed-form analytical formula for the optimal learning rate and complex eigenvalue boundaries of B_j for K >= 2 (Appendix B & C.1, p. 13, 17), and its continuous Beta-Averaged Momentum analog requires storing the full gradient history (Appendix F, p. 21).
⚠️ Curated Limitations & Negative Results
- Requires storing K velocity vectors of full parameter dimension (e.g., K = 3 or K = 4), multiplying momentum state memory by K compared to classical or Nesterov momentum.
- Because the (K + 1)-th order finite difference dynamics do not correspond to a standard 2nd-order Hamiltonian ODE discretization, closed-form optimal learning rates for general K >= 2 are unavailable and Beta-Averaged Momentum requires full gradient history.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ArtificialNeuralNetwork
Artificial Neural Network
|
related_match | AggMo is a drop-in momentum optimization method for training deep autoencoders, convolutional neural networks, and recurrent language models. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
r606-primary-pdf
|
James Lucas, Shengyang Sun, Richard Zemel, Roger Grosse
|
primary paper
full_text
|
ID: https://dblp.org/rec/conf/iclr/LucasSZG19
sha256:b69c87fd4b1f…
|
r606-arxiv-abs
|
James Lucas, Shengyang Sun, Richard Zemel, Roger Grosse
|
primary paper
abstract_only
|
ID: https://arxiv.org/abs/1804.00325
sha256:a2a5fdd0f834…
|
r606-metadata-cr
|
James Lucas, Shengyang Sun, Richard Zemel, Roger Grosse
|
primary paper
abstract_only
|
ID: https://dblp.org/rec/conf/iclr/LucasSZG19
sha256:b30cded505c2…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T11:29:00Z
Updated: 2026-10-19T11:29:00Z
Source File:
data/curated/AggMoAggregatedMomentumPassiveDampingOptimization.yaml
· Schema:
curated_concept.yaml
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories