Home / Curated / Algorithms / Layer-wise adaptive rate scaling

Layer-wise adaptive rate scaling

Algorithms AIMech:LayerWiseAdaptiveRateScaling DRAFT

A gradient optimization method that multiplies each selected layer or parameter block direction by a scalar based on its current weight L2 norm relative to its gradient L2 norm, together with a trust coefficient and global learning rate. Its stated weight-decay extension scales a combined gradient-and-decay direction using a sum-of-norms denominator and can be composed with momentum.

Identity Notes & Concept Scope Compared all 442 pinned AIO and 563 maintained IDs, names, aliases and scoped synonyms. The initial bare LARS candidate alias matched AIMech:LeastAngleRegression; its complete record and proposal context establish an unrelated equiangular regression algorithm. Removed that ambiguous alias. The expanded optimizer identity then had no normalized match. Ignored- and hidden-inclusive fixed-string searches of the recovery, older discovery and original Documents checkouts returned 330, 262 and 267 lines, including the original ignored virtual environment. Canonical hits identify the regression acronym, author surnames and pending optimizer leads; long historical reports and generated mirrors were classified but not all reread. Read all ten semantic AIO rows and 85 native definitions, three complete annotated AIO nodes with incident edges and generated YAML, and six full native context records. Layer-wise weight/gradient-norm scaling differs from ordinary SGD, coordinate-wise Adam/RMSprop moments, decoupled AdamW decay, matrix-orthogonalized Muon and least-angle regression. This record does not assign the same identity to every later LARS/LARC/LAMB implementation.

Claims & Grounded Evidence

6 Claims

Mechanism #block-scalar-adaptation

For a selected layer or parameter block with nonzero weight and gradient L2 norms, the basic local multiplier is lambda_t=eta*norm(w_t)/norm(g_t), where g_t is the current minibatch objective gradient and eta is a positive trust coefficient. A global learning-rate schedule gamma_t supplies the fresh descent increment gamma_t*lambda_t*g_t. This is one scalar per chosen block, rather than a coordinate-wise accumulated gradient-moment denominator or a new network layer.

supports Source: Large Batch Training of Convolutional Networks · Locator: Section 4, Equations 4–5, PDF/printed p. 4; comparison with Adam and RMSprop, p. 2

Mechanism #coupled-decay-extension

For nonnegative coupled weight decay beta, Equation 6 uses lambda_t=eta*norm(w_t)/(norm(g_t)+beta*norm(w_t)) and scales the combined direction g_t+beta*w_t. The denominator is the sum of two norms with the decay coefficient, not norm(g_t+beta*w_t). Decay therefore participates in the adaptive multiplier and momentum direction; this is not an independent decoupled shrinkage step.

supports Source: Large Batch Training of Convolutional Networks · Locator: Section 4, Equation 6 and Algorithm 1, PDF/printed p. 4

Mechanism #momentum-and-schedule

A consistently indexed momentum composition is v_(t+1)=m*v_t+gamma_t*lambda_t*(g_t+beta*w_t), followed by w_(t+1)=w_t-v_(t+1), starting with zero velocity. Here gamma_t denotes the schedule value applied to the current update. This record retains eta from Equations 5–6: Algorithm 1 lists eta but omits it from its local-multiplier line, computes gamma_t but uses gamma_(t+1), and omits an explicit loop-counter increment. Polynomial decay and warm-up are schedule choices in the examples, not the identity of the norm-ratio adaptation.

supports Source: Large Batch Training of Convolutional Networks · Locator: Section 4, Equations 4–6 and Algorithm 1, p. 4; Section 5 and footnote 6, p. 5; ResNet training setup, p. 7

Limitation #relative-increment-boundary

With no decay, nonzero norms and nonnegative gamma_t, the fresh increment has norm gamma_t*eta*norm(w_t) and is unchanged by multiplying g_t by a positive scalar. With beta>=0 and a positive denominator, the triangle inequality bounds the fresh combined-direction increment by that same quantity. These are algebraic properties of the stated formula, not a bound on the whole accumulated momentum displacement, a general stability theorem, or a solution of a trust-region subproblem.

supports Source: Large Batch Training of Convolutional Networks · Locator: Section 4, Equations 4–6 and momentum composition, PDF/printed p. 4; mathematical consequences qualified by the stated conditions

Limitation #zero-norm-and-implementation-boundaries

The primary formula leaves a zero denominator undefined and supplies no epsilon or fallback policy. At zero weight norm with a positive denominator it gives zero fresh increment; existing momentum can still move the block. Bias/normalization exclusions, parameter grouping, clipping and zero-norm handling must be specified by a concrete implementation and cannot be inferred from the method name. The source discusses weights and biases separately, but does not establish a universal exclusion rule.

supports Source: Large Batch Training of Convolutional Networks · Locator: Section 4, Table 2, Equations 5–6 and Algorithm 1, p. 4; separate weight/bias local-rate plots, Figure 2, p. 5

Limitation #scope-and-evidence-limits

The primary study targets large-batch convolutional-network training with synchronous minibatch gradients, global schedules, momentum and model-specific experimental settings. Its reported accuracy depends on architecture, batch size, schedule and training duration. The norm-ratio rule does not by itself guarantee convergence, eliminate every vanishing/exploding-gradient problem, preserve accuracy at arbitrary batch size, or establish distributed speedup. This record curates the mechanism without importing a numerical benchmark or claiming a source training reproduction.

supports Source: Large Batch Training of Convolutional Networks · Locator: Abstract and Sections 1–2, pp. 1–2; Sections 3–7, Tables 1–5 and Figures 1–5, pp. 3–7

⚠️ Curated Limitations & Negative Results

Bare LARS is not asserted as an alias because AIMech already uses it for Least-angle regression, an unrelated equiangular regression algorithm. This record uses the expanded optimizer name.
The mechanism follows Equations 5–6 of arXiv1708.03888v3 and an explicitly consistent current-step indexing. The displayed Algorithm 1 omits eta in one line and has schedule/counter inconsistencies; no literal pseudocode reproduction or author implementation audit is claimed.
A zero-norm fallback, epsilon, clipping rule, bias/normalization exclusion and precise parameter grouping are implementation decisions not specified by the inspected formula. Later similarly named optimizer variants require separate source and identity assessments.
The fresh-increment norm statement does not bound momentum accumulation or establish universal stability, scale invariance of an entire network, convergence or generalization.
Source prose, printed ratios and experimental tables contain inconsistencies retained in the discovery notes. This record makes no universal no-accuracy-loss assertion or numerical performance claim.
The complete eight-page September 13, 2017 technical-report preprint was inspected. Later Berkeley/ICLR versions and the linked author implementation were not treated as identical evidence. No training, source software execution or empirical evaluation occurred.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:TrainingStrategy` TrainingStrategy	related_match	The optimizer directly implements neural-network parameter training, the central context named by TrainingStrategy. The pinned AIO definition uses a preprocessing genus and lists learning-task families, so it is not treated as an algorithm superclass or exact identity. The mapping states a contextual training relationship; it does not make this optimizer a preprocessing step.

Evidence Sources & Primary Literature

1 Source

Source ID	Publication / Document	Type / Access	Reference & Checksum
`you2017`	Large Batch Training of Convolutional Networks Yang You, Igor Gitman, Boris Ginsburg	primary paper full_text	ID: arXiv:1708.03888v3 sha256:8012e213ce51…

Curation Provenance

Curator: Codex

Method: ai_assisted

Model: GPT-6

Created: 2026-10-13T02:30:20Z

Updated: 2026-10-13T02:30:20Z

Source File: data/curated/LayerWiseAdaptiveRateScaling.yaml