Home / Curated / Algorithms / Complete layer-wise adaptive rate scaling

Complete layer-wise adaptive rate scaling

Algorithms AIMech:CompleteLayerWiseAdaptiveRateScaling DRAFT

Aliases: CLARS

A layer-wise gradient optimization method whose published rate formula divides the current parameter-block L2 norm by the average L2 norm of individual-example gradients in a minibatch, multiplied by a global rate and trust coefficient. It applies the resulting scalar to the minibatch mean gradient, reducing the basic LARS rate when individual gradient directions disagree.

Identity Notes & Concept Scope Compared all 442 pinned AIO and 564 native IDs, names, aliases and scoped synonyms; no normalized identity match. Ignored- and hidden-inclusive fixed-string searches of the recovery, older discovery and original Documents checkouts returned 10, 0 and 0 lines. All ten hits were read and identify the pending CLARS lead or review scope in the preceding round, not a record. A semantic search covered ten AIO rows and 87 native definitions: ten AIO and 85 native rows were verified unchanged from the prior reading, with the two additional definitions read. Three complete AIO node/edge/generated contexts were verified unchanged; six full native context records were read or verified against prior full readings. The per-example mean-norm denominator distinguishes CLARS from the norm-of-mean LARS rule, ordinary SGD and coordinatewise Adam/RMSprop updates. Warm-up in Hamiltonian step-size adaptation serves a different sampler-control identity. The published equation and later or approximate implementations are explicitly distinguished.

Claims & Grounded Evidence

7 Claims

Mechanism #mean-individual-norm-rate

For a minibatch of B examples and block k, let gbar_k=(1/B)*sum_i g_(i,k) and D_k=(1/B)*sum_i norm(g_(i,k)), with L2 norms and gradients evaluated at the current parameters. When D_k>0, Equation 5 sets gamma_k=gamma_scale*eta*norm(w_k)/D_k and the fresh block descent increment is gamma_k*gbar_k. The paper uses gamma_scale=gamma_base*B/B_base and a selected positive eta. D_k is an average of unsquared norms, not norm(gbar_k), a mean squared norm, or a coordinate-wise running moment.

supports Source: Large Batch Optimization for Deep Learning Using New Complete Layer-Wise Adaptive Rate Scaling · Locator: Equations 1, 4–5 and factorization, PDF p. 2 / p. 7884

Mechanism #coherence-and-relative-bound

For a nonzero mean gradient, the published rate equals the basic no-decay LARS rate multiplied by rho_k=norm(gbar_k)/D_k. Triangle inequality gives 0<=rho_k<=1 when D_k>0; with nonnegative global rate the fresh increment norm is gamma_scale*eta*norm(w_k)*rho_k. Positively aligned individual gradients give rho_k=1 even when their magnitudes differ. This norm ratio reflects directional agreement but is not itself an exact gradient variance or covariance estimate. Rescaling every individual gradient by the same positive scalar preserves the ideal fresh increment.

supports Source: Large Batch Optimization for Deep Learning Using New Complete Layer-Wise Adaptive Rate Scaling · Locator: Equation 5, its following factorization and variance discussion, PDF p. 2 / p. 7884; algebraic consequences under the stated conditions

Mechanism #nesterov-composition

The published Equation 4 composes the block-scaled mean gradients with v_(t+1)=w_t-sum_k U_k*gamma_k*gbar_k and w_(t+1)=v_(t+1)+beta*(v_(t+1)-v_t), where U_k places a block into the parameter vector and 0<=beta<1. Here v is an auxiliary parameter iterate, not the accumulated gradient velocity in every optimizer library. The method requires an initialization convention for the paired iterates. The fresh-increment norm statement does not bound the whole extrapolated displacement, and the equation does not automatically establish equivalence to the inspected code momentum convention.

supports Source: Large Batch Optimization for Deep Learning Using New Complete Layer-Wise Adaptive Rate Scaling · Locator: Preliminaries, Equations 2 and 4, PDF p. 2 / p. 7884
supports Source: Author largebatch implementation: lars_optimizer.py · Locator: LARSOptimizer.step, momentum buffer and scaled update, lines 118–132

Limitation #zero-cases-and-pseudocode

If the mean gradient is zero but D_k>0, direct Equation 5 has a finite multiplier and zero fresh increment; the factorization through a LARS division by norm(gbar_k) must not be evaluated literally at that point. If all individual gradients vanish then D_k=0 and the paper formula supplies no fallback. Zero weights with D_k>0 give zero fresh increment, although extrapolation can still move parameters. Algorithm 1 step 4 prints squared gradient norms while step 5 refers to the unsquared-norm Equation 5; this record follows Equation 5 and preserves that discrepancy.

supports Source: Large Batch Optimization for Deep Learning Using New Complete Layer-Wise Adaptive Rate Scaling · Locator: Equation 5 and factorization, PDF p. 2 / p. 7884; Algorithm 1 steps 4–6, PDF p. 3 / p. 7885

Limitation #implementation-approximation-boundary

The exact denominator requires individual-example gradient norms and cannot be recovered from the aggregate gradient alone. The paper describes a smaller-sample approximation. At the inspected author-code revision, the ImageNet script instead averages norms of gradients of eight-example minibatch losses over a configurable number of passes, defaulting to 64; this is not literally the mean of 512 individual-example gradient norms. The optimizer also adds coupled decay before its gradient norm, uses numerical offsets and a capped multiplier, supports parameter exclusions, and applies its rate outside the momentum buffer. These are versioned implementation choices rather than a literal execution of Equations 4–5.

supports Source: Large Batch Optimization for Deep Learning Using New Complete Layer-Wise Adaptive Rate Scaling · Locator: Comparison with Related Methods, PDF p. 5 / p. 7887
supports Source: Author largebatch implementation: lars_optimizer.py · Locator: LARSOptimizer.step, lines 95–132
supports Source: Author largebatch implementation: main_imagenet_lars.py · Locator: Arguments and optimizer setup, lines 46–53 and 96–105; M_loader, lines 154–156; compute_M, lines 364–387

Limitation #convergence-conditions

The published analysis assumes block and global Lipschitz gradients, a layer-wise stochastic-gradient variance bound and specified step-size restrictions. Its stated theorems analyze gradient and Nesterov methods with rates gamma_k=gamma/L_k and obtain qualified stationarity bounds. The empirical norm ratio is not an identity for an inverse Lipschitz constant, and those conditional results do not by themselves certify every random data-dependent Equation 5 rate, approximation, clipping rule or source-code variant. No unrestricted convergence or universal removal of warm-up follows solely from the formula.

supports Source: Large Batch Optimization for Deep Learning Using New Complete Layer-Wise Adaptive Rate Scaling · Locator: Assumptions 1–2 and Theorem 1, PDF p. 3 / p. 7885; Theorem 2, Corollaries 1–3 and discussion, PDF p. 4 / p. 7886

Limitation #experimental-and-schedule-scope

The method was proposed for large-batch neural-network training and studied with selected architectures, data, schedules, trust coefficients and denominator approximations. Warm-up-free examples do not establish that warm-up is unnecessary in every setting or that CLARS always improves accuracy or wall-clock performance. The inspected ImageNet code retains a configurable warm-up schedule even with its layer-wise flag. This record curates the algorithm and source-version boundaries without importing numerical benchmark results or claiming to reproduce training.

supports Source: Large Batch Optimization for Deep Learning Using New Complete Layer-Wise Adaptive Rate Scaling · Locator: Experimental Results and comparisons, Figures 1–4 and Table 1, PDF pp. 5–7 / pp. 7887–7889
supports Source: Author largebatch implementation: main_imagenet_lars.py · Locator: Warm-up argument, line 46; schedule call, lines 208–221; poly_lr_rate, lines 331–341

⚠️ Curated Limitations & Negative Results

This record follows the unsquared individual-gradient norm denominator in published Equation 5. The squared-norm line in Algorithm 1 and differing experimental approximations are disclosed rather than silently treated as identical.
The ideal formula has no specified all-zero denominator fallback. Epsilon, clipping, decay, parameter grouping, excluded tensors and paired-iterate initialization require implementation-specific choices.
The coherence ratio is not an exact variance estimator; a fresh-increment bound is not a bound on extrapolated momentum or a general convergence result. The stated theorems have stronger step-size and smoothness conditions than an arbitrary adaptive ratio.
The author code was inspected statically at an August 22, 2020 revision. Its small-batch norm approximation, coupled decay, offsets, cap, parameter exclusions, non-Nesterov script setting and optional warm-up are not assumed identical to the 2021 paper equations or experimental configuration.
The published experiment prose, figure labels and table captions contain discrepancies preserved in the discovery notes. No numerical accuracy, speed, generalization or universal no-warm-up claim is imported.
All eight pages of the 2021 proceedings paper and selected versioned author files were read. Earlier arXiv full text, dissertation proofs, model implementations and complete experimental command provenance remain unverified. No downloaded code was executed and no model was trained.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:TrainingStrategy` TrainingStrategy	related_match	The method directly controls parameter updates during neural-network training, the central context named by TrainingStrategy. Its pinned preprocessing genus and learning-task examples do not make it an optimizer superclass or equivalent identity. The mapping expresses contextual training relatedness without classifying CLARS as preprocessing.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`huo2021`	Large Batch Optimization for Deep Learning Using New Complete Layer-Wise Adaptive Rate Scaling Zhouyuan Huo, Bin Gu, Heng Huang	primary paper full_text	ID: doi:10.1609/aaai.v35i9.16962 sha256:c69e17511264…
`authoroptimizer`	Author largebatch implementation: lars_optimizer.py slowbull/largebatch contributors	official documentation documentation	ID: slowbull/largebatch@ea8880af035222de0cbebcfa4d81d100a61ffb5a:lars_optimizer.py sha256:3d2b57d61c42…
`authorimagenet`	Author largebatch implementation: main_imagenet_lars.py slowbull/largebatch contributors	official documentation documentation	ID: slowbull/largebatch@ea8880af035222de0cbebcfa4d81d100a61ffb5a:main_imagenet_lars.py sha256:45da90334a41…

Curation Provenance

Curator: Codex

Method: ai_assisted

Model: GPT-6

Created: 2026-10-13T02:51:49Z

Updated: 2026-10-13T02:51:49Z

Source File: data/curated/CompleteLayerWiseAdaptiveRateScaling.yaml