Home / Curated / Algorithms / SWATS

SWATS

Algorithms AIMech:SWATS DRAFT

Aliases: Switching from Adam to SGD Optimization Strategy, Gradient-Subspace Projected Adam-to-SGD Hybrid Optimizer, Automated Adam-to-SGD Switchover Optimization

A hyperparameter-free hybrid stochastic optimization strategy (Switching from Adam to SGD) that begins training in an Adam phase, estimates a scalar SGD step size gamma_k = (p_k^T p_k) / (-p_k^T g_k) at each step k by solving the projection equation proj_{-gamma_k g_k} p_k = p_k between the Adam step p_k and negative stochastic gradient -g_k, tracks an exponential moving average lambda_k = beta_2 lambda_{k-1} + (1 - beta_2) gamma_k reusing Adam's second-moment decay rate beta_2, and permanently switches to SGD (or SGDM with momentum beta_1 and scale (1 - beta_1) Lambda) with scalar learning rate Lambda = lambda_k / (1 - beta_2^k) as soon as |lambda_k / (1 - beta_2^k) - gamma_k| < epsilon.

Identity Notes & Concept Scope Distinct from AdaBound/AMSBound (AIMech:AdaBoundAndAMSBound, which smoothly pinches coordinate-wise step sizes via scheduled upper and lower bounds parameterized by gamma) and from Padam (AIMech:Padam, which maintains a constant partial-adaptivity power p in (0, 1/2] throughout training). SWATS executes a discrete phase switch from Adam to uniform-step-size SGD(M) where both the switchover iteration and the post-switch SGD learning rate Lambda are inferred automatically from the projection of -gamma_k g_k onto p_k.

Claims & Grounded Evidence

4 Claims

Mechanism #swats-non-orthogonal-projection-and-rayleigh-quotient-identity

Given stochastic gradient g_k and Adam step p_k != 0 with p_k^T g_k < 0, SWATS solves proj_{-gamma_k g_k} p_k = -gamma_k ((g_k^T p_k) / (p_k^T p_k)) p_k = p_k in closed form to obtain gamma_k = (p_k^T p_k) / (-p_k^T g_k); when beta_1 = 0 so p_k = -H_k g_k with diagonal preconditioner H_k > 0, gamma_k > 0 equals the reciprocal of the Rayleigh quotient of H_k^{-1} with respect to p_k and avoids the systematic step-size underestimation of the orthogonal projection proj_{p_k}(-g_k).

supports Source: Improving Generalization Performance by Switching from Adam to SGD · Locator: Section 2.1 (Learning rate for SGD after the switch) and Figure 3, p. 3-5
supports Source: Improving Generalization Performance by Switching from Adam to SGD (Crossref metadata) · Locator: Bibliographic metadata for Keskar & Socher 2017

Scope #swats-bias-corrected-ema-switchover-and-column-space-scope

SWATS smooths gamma_k via lambda_k = beta_2 lambda_{k-1} + (1 - beta_2) gamma_k and triggers a permanent switch to SGD(M) with learning rate Lambda = lambda_k / (1 - beta_2^k) and momentum update v_k = beta_1 v_{k-1} + g_k, w_k = w_{k-1} - (1 - beta_1) Lambda v_k when |lambda_k / (1 - beta_2^k) - gamma_k| < epsilon (k > 1), transitioning from non-uniform diagonal scaling to isotropic updates that remain within the gradient column space.

supports Source: Improving Generalization Performance by Switching from Adam to SGD · Locator: Section 2.2 (Switchover Point, Equation 4), Algorithm 1, and Section 4, p. 4-5, 8

Application #swats-vision-and-language-modeling-generalization-applications

SWATS closes the test-error generalization gap between Adam and tuned SGD(M) on CIFAR-10 and CIFAR-100 across ResNet-32, DenseNet, PyramidNet, and SENet as well as ResNet-18 on Tiny-ImageNet, while matching Adam's validation perplexity on Penn Treebank and WikiText-2 word-level language modeling with AWD-LSTM and AWD-QRNN.

supports Source: Improving Generalization Performance by Switching from Adam to SGD · Locator: Section 3 (Numerical Results), Table 1 and Figures 4-7, p. 5-9

Limitation #swats-hard-switch-transient-degradation-and-irreversibility-limitation

Because SWATS executes an abrupt one-way hard switch from per-coordinate adaptive scaling to a single scalar SGD learning rate and resets the momentum buffer v_k = 0 at switchover, it can induce a sharp temporary drop in test accuracy immediately after the switch (such as dropping from 80% to 52% on Tiny-ImageNet before recovering) and cannot switch back if gradient scales later become ill-conditioned.

supports Source: Improving Generalization Performance by Switching from Adam to SGD · Locator: Section 3 (p. 6, Figure 5) and Section 4 (Discussion and Conclusion, p. 8)

⚠️ Curated Limitations & Negative Results

Executes a hard one-way switch with momentum buffer reset v_k = 0, which can cause a significant temporary drop in accuracy immediately after the switchover epoch.
Relies on an unnormalized scalar convergence criterion |lambda_k / (1 - beta_2^k) - gamma_k| < epsilon that may trigger late on high-variance recurrent language modeling tasks.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	SWATS is a hybrid optimization method designed to combine Adam's fast initial progress with SGD's generalization performance when training deep neural networks.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r615-primary-pdf`	Improving Generalization Performance by Switching from Adam to SGD Nitish Shirish Keskar, Richard Socher	primary paper full_text	ID: https://dblp.org/rec/journals/corr/abs-1712-07628.xml sha256:a5c0875c1c61…
`r615-arxiv-abs`	Improving Generalization Performance by Switching from Adam to SGD (arXiv abstract) Nitish Shirish Keskar, Richard Socher	primary paper abstract_only	ID: https://arxiv.org/abs/1712.07628 sha256:9e92c3a4b805…
`r615-metadata-cr`	Improving Generalization Performance by Switching from Adam to SGD (Crossref metadata) Nitish Shirish Keskar, Richard Socher	primary paper abstract_only	ID: https://dblp.org/rec/journals/corr/abs-1712-07628.xml sha256:7311fb071ea7…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T11:38:00Z

Updated: 2026-10-19T11:38:00Z

Source File: data/curated/SWATS.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories