AdaBound and AMSBound Dynamic Learning-Rate Bounds Optimization
Algorithms AIMech:AdaBoundAndAmsBoundDynamicLearningRateBoundsOptimization DRAFT
Aliases: AdaBound, AMSBound, Adaptive Gradient Methods with Dynamic Bound of Learning Rate, Dynamic Learning-Rate Bound Adaptive Moment Estimation
AdaBound and AMSBound are first-order adaptive stochastic optimization algorithms (Luo, Xiong, Liu, & Sun, ICLR 2019) that eliminate both extremely large and extremely small late-stage learning rates in Adam and AMSGrad by clipping the element-wise adaptive learning rate alpha / sqrt(V_t) to a pair of dynamic monotone bound functions [eta_l(t), eta_u(t)]: eta_hat_t = Clip(alpha / sqrt(V_t), eta_l(t), eta_u(t)), eta_t = eta_hat_t / sqrt(t), and x_{t+1} = Pi_{F, diag(eta_t^{-1})}(x_t - eta_t odot m_t), where V_t = diag(v_t) for AdaBound and V_t = diag(v_hat_t) with v_hat_t = max(v_hat_{t-1}, v_t) for AMSBound, and where the non-decreasing lower bound eta_l(t) (initialized at 0) and non-increasing upper bound eta_u(t) (initialized at infinity) both converge smoothly to a constant final step size alpha* as t -> infinity, inducing a continuous transition from adaptive gradient methods early in training to SGD (with momentum) asymptotically.
Identity Notes & Concept Scope
Distinct from Adam (Kingma & Ba, 2015), which sets eta_l(t) = 0 and eta_u(t) = infinity and can fail to converge on convex problems due to either huge or tiny learning rates (Theorems 1-3); distinct from AMSGrad (Reddi et al., 2018), which enforces non-increasing step sizes via v_hat_t = max(v_hat_{t-1}, v_t) to curb large learning rates but does not bound tiny learning rates from below; distinct from SWATS (Keskar & Socher, 2017), which performs a discrete hard switch from Adam to SGD at a thresholded step rather than a continuous dynamic-bound transformation; and distinct from Padam (Chen et al., 2020), which scales the second-moment denominator exponent to v_hat_t^p with p in (0, 1/2] rather than clipping step sizes to time-varying bounds [eta_l(t), eta_u(t)].
Claims & Grounded Evidence
4 Claims
Mechanism
#theorem1-nonconvergence-of-adam-from-tiny-learning-rates
Theorem 1 (Section 3, p. 4 & Appendix B, Eqs. 2-3, p. 12-13) proves that for the 1D online convex optimization sequence over F = [-2, 2] with f_t(x) = -100 for x < 0, f_t(x) = -x for 0 <= x <= 1 and t mod C = 1, f_t(x) = 2x for 0 <= x <= 1 and t mod C = 2, and 0 otherwise, where C in N satisfies 5 * beta_2^{C-2} <= (1 - beta_2)/(4 - beta_2) (Eq. 3), Adam with beta_1 = 0 and any initial step size alpha > 0 satisfies V_{Ct+1} - 4 V_{Ct} >= (4 - beta_2)[(1 - beta_2)/(4 - beta_2) - 5 * beta_2^{C-2}] >= 0, keeping x_t >= 0 for all t in N and suffering non-vanishing average regret R_T / T >= 100 as T -> infinity because the corrective gradient +2 at t mod C = 2 is scaled down by a larger second-moment accumulator than the wrong-direction gradient -1 at t mod C = 1.
Mechanism
#theorem4-and-theorem5-online-convex-regret-bounds
Theorems 4 and 5 and Corollary 4.1 (Section 4, p. 5 & Appendix E-F, p. 14-17) prove that under beta_1 = beta_{11}, beta_{1t} <= beta_1, beta_1 / sqrt(beta_2) < 1, monotone bounds eta_l(t+1) >= eta_l(t) > 0 and eta_u(t+1) <= eta_u(t) converging to alpha* with L_infty = eta_l(1) and R_infty = eta_u(1), bounded diameter ||x - y||_infty <= D_infty, and bounded gradients ||nabla f_t(x)||_2 <= G_2, both AdaBound (Algorithm 2) and AMSBound (Algorithm 3) satisfy R_T <= (D_infty^2 sqrt(T) / (2(1 - beta_1))) sum_{i=1}^d eta_hat_{T,i}^{-1} + (D_infty^2 / (2(1 - beta_1))) sum_{t=1}^T sum_{i=1}^d beta_{1t} eta_{t,i}^{-1} + (2 sqrt(T) - 1) R_infty G_2^2 / (1 - beta_1) = O(sqrt(T)) when beta_{1t} = beta_1 lambda^{t-1} or beta_{1t} = beta_1 / t.
Application
#canonical-rational-bound-functions-and-deep-network-generalization
Using the canonical bound functions eta_l(t) = (1 - 1 / ((1 - beta)t + 1)) alpha* and eta_u(t) = (1 + 1 / ((1 - beta)t)) alpha* with default alpha = 0.001, alpha* = 0.1, beta_1 = 0.9, and beta_2 = 0.999 (Section 4-5, p. 6-9; Appendix G-H, p. 17-19), AdaBound and AMSBound match the early convergence speed of Adam/AMSGrad while improving CIFAR-10 test accuracy by ~2% over Adam on DenseNet-121 and surpassing SGDM by ~1% on ResNet-34, and reduce Penn Treebank test perplexity by over 2.8% relative to Adam on 3-layer LSTM language models while remaining robust across alpha* in [0.001, 1] and beta in [beta_1, beta_2].
Limitation
#modest-gains-on-shallow-models-and-bound-schedule-specification
Section 5.4-6 (p. 8-9) notes that on simple shallow architectures such as a 1-layer perceptron on MNIST or a 1-layer LSTM on Penn Treebank, the test accuracy or perplexity improvement of AdaBound and AMSBound over Adam and SGD is modest (~1.1% on 1-layer LSTM), that on recurrent language modeling tasks AdaBound and AMSBound do not match Adam's very earliest-epoch drop in perplexity, and that the algorithms require specifying two dynamic bound schedules eta_l(t) and eta_u(t) together with a target SGD step size alpha*.
⚠️ Curated Limitations & Negative Results
- Improvements over vanilla Adam and SGD are modest on shallow architectures such as a 1-layer MNIST perceptron or a 1-layer LSTM compared with deeper ResNet, DenseNet, and 3-layer LSTM models (Section 5.2, 5.4, 6, p. 7-9).
- On Penn Treebank LSTM language modeling, AdaBound and AMSBound exhibit smoother learning curves and lower final perplexity than Adam and SGD, but do not match Adam's rapid perplexity drop during the very earliest training epochs (Section 5.4, Figure 4, p. 8).
- Requires specifying functional forms for the lower and upper dynamic bounds eta_l(t) and eta_u(t) and the asymptotic SGD target step size alpha* (Section 4, 6, p. 6, 9).
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ArtificialNeuralNetwork
Artificial Neural Network
|
related_match | AdaBound and AMSBound are designed to train feedforward, convolutional (DenseNet-121, ResNet-34), and recurrent (LSTM) deep neural networks with rapid initial progress and strong out-of-sample generalization. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
r608-primary-pdf
|
Liangchen Luo, Yuanhao Xiong, Yan Liu, Xu Sun
|
primary paper
full_text
|
ID: https://dblp.org/rec/conf/iclr/LuoXLS19
sha256:d2bca40bc2d3…
|
r608-arxiv-abs
|
Liangchen Luo, Yuanhao Xiong, Yan Liu, Xu Sun
|
primary paper
abstract_only
|
ID: https://arxiv.org/abs/1902.09843
sha256:6e646d4b8652…
|
r608-metadata-cr
|
Liangchen Luo, Yuanhao Xiong, Yan Liu, Xu Sun
|
primary paper
abstract_only
|
ID: https://dblp.org/rec/conf/iclr/LuoXLS19
sha256:4c9119222166…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T11:31:00Z
Updated: 2026-10-19T11:31:00Z
Source File:
data/curated/AdaBoundAndAmsBoundDynamicLearningRateBoundsOptimization.yaml
· Schema:
curated_concept.yaml
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories