YellowFin Automatic Momentum and Learning Rate Tuner
Algorithms AIMech:YellowFinAutomaticMomentumAndLearningRateTuner DRAFT
Aliases: YellowFin, YellowFin Optimizer, YellowFin Momentum Tuner, Closed-Loop YellowFin
An automatic hyperparameter tuning algorithm for Polyak heavy-ball momentum stochastic gradient descent (x_{t+1} = x_t - alpha_t grad f_{S_t}(x_t) + mu_t (x_t - x_{t-1})) that estimates extremal generalized curvatures (h_max, h_min) over a sliding window of squared gradient norms h_t = ||g_t||^2, total coordinate-wise gradient variance C = 1^T (g^2_bar - (g_bar)^2), and local quadratic distance to optimum D from exponential moving averages of ||g_t|| / ||g_t||^2, and at each iteration solves the SingleStep noisy-quadratic surrogate minimization min_{mu_t, alpha_t} (mu_t D^2 + alpha_t^2 C) subject to mu_t >= ((sqrt(h_max / h_min) - 1) / (sqrt(h_max / h_min) + 1))^2 and alpha_t = (1 - sqrt(mu_t))^2 / h_min in closed form via Vieta's cubic substitution, optionally adjusting algorithmic momentum via a negative-feedback control loop in asynchronous settings.
Identity Notes & Concept Scope
Distinct from per-coordinate adaptive learning-rate methods (AdaGrad, RMSProp, Adam) and from Lookahead or SWATS: YellowFin tunes a single global learning rate alpha_t and a single global Polyak momentum parameter mu_t at every step by enforcing the momentum operator's robust-region invariants (1 - sqrt(mu_t))^2 <= alpha_t h <= (1 + sqrt(mu_t))^2 across the dynamic range nu = h_max / h_min of generalized curvatures so that the bias operator A_t has homogeneous spectral radius sqrt(mu_t) and the variance operator B_t has spectral radius mu_t.
Claims & Grounded Evidence
4 Claims
Mechanism
#c-r620-robust-region-homogeneous-spectral-radii
For any scalar or multidimensional objective whose generalized curvature eigenvalues h satisfy the robust-region inequality (1 - sqrt(mu))^2 <= alpha h <= (1 + sqrt(mu))^2, the Polyak momentum bias operator A_t has complex-conjugate eigenvalues of magnitude sqrt(mu) so its spectral radius is rho(A_t) = sqrt(mu) independent of alpha and h (Lemmas 3 and 7), and under the exact same condition the 3x3 stochastic variance operator B has spectral radius rho(B) = mu (Lemmas 6 and 10), reducing the expected squared distance surrogate to mu^t ||x_0 - x*||^2 + (1 - mu^t) alpha^2 C / (1 - mu).
Mechanism
#c-r620-singlestep-cubic-closed-form-and-oracles
Substituting x = sqrt(mu) in [0, 1) and alpha = (1 - x)^2 / h_min into SingleStep (Eq. 15) yields the unimodal quartic objective p(x) = x^2 D^2 + (1 - x)^4 C / h_min^2, whose unique stationary point x_p in (0, 1) solves the cubic equation 2 x D^2 - (4 C / h_min^2)(1 - x)^3 = 0 in closed form via Vieta's substitution, giving the exact constrained optimizer sqrt(mu_t) = max(x_p, (sqrt(h_max / h_min) - 1) / (sqrt(h_max / h_min) + 1)) from O(d) gradient-only measurement oracles CurvatureRange, Variance, and Distance.
Application
#c-r620-closed-loop-asynchronous-momentum-control
In asynchronous-parallel training with staleness tau, stale gradients induce extra implicit momentum so total system momentum mu_T exceeds algorithmic momentum mu; Closed-loop YellowFin (Algorithm 5) estimates mu_hat_T = median((x_{t-tau} - x_{t-tau-1} + alpha grad f_{S_{t-tau-1}}(x_{t-tau-1})) / (x_{t-tau-1} - x_{t-tau-2})) and applies negative feedback mu <- mu + gamma (mu* - mu_hat_T), achieving up to 2.69x speedup over Adam on 16-worker asynchronous ResNet training.
Limitation
#c-r620-non-smooth-exploding-gradient-limitation
Because YellowFin approximates directional Hessian curvature by squared stochastic gradient norms h_t = ||g_t||^2 under a negative log-likelihood Fisher assumption, abrupt steep slopes and exploding gradients in non-stationary recurrent or sequence-to-sequence landscapes can corrupt the upper curvature envelope h_max unless stabilized by capping envelope growth at 100 * h_max and clipping gradients at threshold sqrt(h_max).
⚠️ Curated Limitations & Negative Results
- Relies on the negative log-probability Fisher approximation h_t = ||g_t||^2 to estimate directional curvature over a sliding window of width w = 20, which is not guaranteed to be an accurate Hessian eigenvalue estimate on general objectives.
- Requires zero-debiasing on log-extremal curvatures, a 10w-step slow-start learning-rate discount, and adaptive gradient clipping at sqrt(h_max) with a 100x envelope growth cap to avoid divergence on non-smooth objectives with exploding gradients.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ArtificialNeuralNetwork
Artificial Neural Network
|
related_match | YellowFin is an automatic momentum and learning-rate tuning algorithm for training deep artificial neural networks in synchronous and asynchronous regimes. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
r620-primary-pdf
|
Jian Zhang, Ioannis Mitliagkas, Christopher Ré
|
primary paper
full_text
|
ID: https://dblp.org/rec/conf/mlsys/ZhangM19
sha256:7b021293dbb7…
|
r620-arxiv-abs
|
Jian Zhang, Ioannis Mitliagkas, Christopher Ré
|
primary paper
abstract_only
|
ID: https://arxiv.org/abs/1706.03471
sha256:6555d1d8fa28…
|
r620-metadata-cr
|
Jian Zhang, Ioannis Mitliagkas, Christopher Ré
|
primary paper
abstract_only
|
ID: https://dblp.org/rec/conf/mlsys/ZhangM19
sha256:217454ab5621…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T11:43:00Z
Updated: 2026-10-19T11:43:00Z
Source File:
data/curated/YellowFinAutomaticMomentumAndLearningRateTuner.yaml
· Schema:
curated_concept.yaml
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories