Home / Curated / Algorithms / ACProp Momentum-Centered Asynchronous Adaptive Gradient Optimization

ACProp Momentum-Centered Asynchronous Adaptive Gradient Optimization

Algorithms AIMech:ACPropMomentumCenteringAsynchronousAdaptiveGradientOptimization DRAFT

Aliases: ACProp, ACProp Optimizer, Asynchronous-Centering-Prop, ACProp Asynchronous-Centering-Prop Optimizer

ACProp (Asynchronous-Centering-Prop) is a first-order adaptive gradient optimization algorithm for stochastic non-convex and online convex optimization that combines second-momentum centering with an asynchronous denominator update rule. At iteration t, given stochastic gradient g_t = grad_x f_t(x_{t-1}) in R^d, ACProp updates the first-moment exponential moving average m_t = beta_1 m_{t-1} + (1 - beta_1) g_t, updates model parameters x_t using the lagged centered second-momentum estimate s_{t-1} from step t - 1 in the denominator (x_t = x_{t-1} - alpha_t g_t / (sqrt(s_{t-1}) + epsilon) or with momentum numerator), and only subsequently updates the centered second momentum s_t = beta_2 s_{t-1} + (1 - beta_2)(g_t - m_t)^2. Decoupling the step-t numerator from the denominator ensures E[g_t / sqrt(s_{t-1} + epsilon) | g_0, ..., g_{t-1}] = E[g_t] / sqrt(s_{t-1} + epsilon), eliminating synchronous step-size bias while preserving non-vanishing centered variance scaling across sparse-gradient regimes.

Identity Notes & Concept Scope Distinct from synchronous uncentered adaptive optimizers (AIMech:Adam, AIMech:RMSProp, AIMech:AMSGrad, AIMech:RAdam), which include g_t^2 inside the step-t denominator v_t; distinct from synchronous centered adaptive optimizers (AIMech:AdaBelief, AIMech:SDProp), which center the second moment via (g_t - m_t)^2 but evaluate s_t synchronously in the step-t denominator; and distinct from asynchronous uncentered optimizers (AIMech:AdaShift), which decorrelate numerator and denominator using uncentered v_{t-n} that decays exponentially to zero during zero-gradient intervals.

Claims & Grounded Evidence

5 Claims

Mechanism #r647-async-unbiased-reddi-convergence

By preconditioning the step-t gradient numerator with the lagged centered second moment s_{t-1} computed solely from history {g_0, ..., g_{t-1}}, ACProp guarantees conditional unbiasedness E[g_t / sqrt(s_{t-1} + epsilon) | g_{0:t-1}] = E[g_t] / sqrt(s_{t-1} + epsilon) and converges on Reddi et al.'s (2018) periodic online convex counterexample for all beta_1, beta_2 in (0, 1) and period P >= 3 (Lemma 3.3 / Lemma A.3).

supports Source: Momentum Centering and Asynchronous Update for Adaptive Gradient Methods · Locator: Section 2.2, p. 3; Section 3.1, Lemma 3.3, pp. 3-5; Appendix A.1, Lemma A.3 & Lemma A.5, pp. 14-17

Mechanism #r647-sparse-gradient-centering-advantage

On periodic sparse-gradient online convex problems where zero gradients occur between opposing non-zero gradient pulses, uncentered asynchronous second moments v_t = beta_2 v_{t-1} decay exponentially toward zero whereas ACProp's centered second moment s_t = beta_2 s_{t-1} + (1 - beta_2)(0 - m_t)^2 remains bounded away from zero when m_t != 0, yielding s^+ / s^- <= v^+ / v^- and a strictly larger hyperparameter convergence region than AdaShift with n = 1 (Lemma 3.4 / Lemma A.6).

supports Source: Momentum Centering and Asynchronous Update for Adaptive Gradient Methods · Locator: Section 3.2, Eq. 2 & Lemma 3.4, pp. 5-6; Appendix A.2, Lemma A.6, Eqs. 50-76, pp. 18-20

Scope #r647-stochastic-nonconvex-oracle-rate

Under L-smoothness and bounded-variance unbiased stochastic gradients with step-size schedule alpha_t = alpha_0 t^{-eta} (eta in [0.5, 1)), ACProp achieves (1/T) sum_{t=1}^T ||grad f(x_t)||^2 = O(1/sqrt(T)) at eta = 0.5 (requiring at most O(delta^{-4}) queries to reach a delta-stationary point, matching the first-order oracle lower bound of Arjevani et al., 2019), and reduces the early-training regret constant from sqrt(mu^2 + sigma^2 + epsilon) to sqrt(sigma^2 + epsilon) when gradient mean |mu| >> sigma (Theorem 4.1, Theorem 4.2, and Eqs. 6-7).

supports Source: Momentum Centering and Asynchronous Update for Adaptive Gradient Methods · Locator: Section 4, Theorems 4.1-4.2 & Eqs. 3-7, pp. 6-7; Appendix B.2, Theorem B.1, pp. 23-25

Application #r647-cnn-gan-transformer-rl-applications

ACProp is applied to train convolutional neural networks on CIFAR-10 and ImageNet (achieving 70.46% Top-1 accuracy with ResNet-18 on ImageNet), Deep Q-Networks on the four-rooms reinforcement learning task, Transformers on IWSLT14/WMT machine translation, and DCGAN, SNGAN, SAGAN, and RLGAN architectures on CIFAR-10.

supports Source: Momentum Centering and Asynchronous Update for Adaptive Gradient Methods · Locator: Section 5, Figures 7-9 & Tables 2-5, pp. 8-9

Limitation #r647-beta2-sensitivity-and-moment-extension-limitation

Empirical sensitivity ablations show that while ACProp is robust across choices of first-moment decay beta_1, its accuracy is more sensitive to the second-moment smoothing hyperparameter beta_2, and extending the theoretical non-convex convergence proof to the momentum-numerator variant (Async-moment-optimizer, Theorem B.3) requires assuming a monotonically non-increasing diagonal preconditioner A_{t+1} <= A_t.

supports Source: Momentum Centering and Asynchronous Update for Adaptive Gradient Methods · Locator: Appendix B.3, Theorem B.3, p. 25; Appendix C.2, Figure 28, pp. 30-31

⚠️ Curated Limitations & Negative Results

ACProp's test accuracy is more sensitive to the second-moment exponential moving average parameter beta_2 than to beta_1 (Appendix C.2, pp. 30-31).
While the vanilla asynchronous update x_{t+1} = x_t - alpha_t A_t g_t achieves the O(1/sqrt(T)) stochastic non-convex convergence bound without monotonicity assumptions on A_t (Theorem 4.1), the theoretical extension to a momentum numerator x_{t+1} = x_t - alpha_t A_t m_t (Theorem B.3) assumes elementwise monotonic preconditioning A_{t+1} <= A_t.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	ACProp is an adaptive first-order gradient optimizer designed for training deep neural networks, generative adversarial networks, transformers, and reinforcement learning agents.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r647-primary-pdf`	Momentum Centering and Asynchronous Update for Adaptive Gradient Methods Juntang Zhuang, Yifan Ding, Tommy Tang, Nicha C. Dvornek, Sekhar Tatikonda, James S. Duncan	primary paper full_text	ID: https://dblp.org/rec/conf/nips/ZhuangDTDTD21 sha256:690397968981…
`r647-arxiv-abs`	Momentum Centering and Asynchronous Update for Adaptive Gradient Methods (arXiv abstract) Juntang Zhuang, Yifan Ding, Tommy Tang, Nicha C. Dvornek, Sekhar Tatikonda, James S. Duncan	primary paper abstract_only	ID: https://arxiv.org/abs/2110.05454 sha256:46c782af07c0…
`r647-metadata-cr`	Momentum Centering and Asynchronous Update for Adaptive Gradient Methods (Crossref metadata) Juntang Zhuang, Yifan Ding, Tommy Tang, Nicha C. Dvornek, Sekhar Tatikonda, James S. Duncan	primary paper abstract_only	ID: https://dblp.org/rec/conf/nips/ZhuangDTDTD21 sha256:f958614aaa5a…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T12:10:00Z

Updated: 2026-10-19T12:10:00Z

Source File: data/curated/ACPropMomentumCenteringAsynchronousAdaptiveGradientOptimization.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories