Home / Curated / Algorithms / Layer-wise adaptive rate control

Layer-wise adaptive rate control

Algorithms AIMech:LayerWiseAdaptiveRateControl DRAFT

Aliases: LARC

A parameter-block learning-rate control rule whose clipping form limits a vanilla stochastic-gradient step by using the smaller of a global learning rate and a trust coefficient times the block parameter-norm to gradient-norm ratio.

Identity Notes & Concept Scope Compared all 442 pinned AIO and 567 native IDs, names, aliases and scoped synonyms; no normalized identity match. Ignored- and hidden-inclusive fixed-string searches of the recovery, older discovery and original Documents checkouts returned 30, 0, 1 lines. All matching lines were read: recovery hits are prior LARS comparisons and pending LARC leads or rendered/review mirrors, while the original ignored environment contains an unrelated APDL lexer command. A semantic search covered ten AIO rows and 91 native definitions: the preceding ten AIO and 90 native rows were verified unchanged, and the new NovoGrad definition was read. Three complete AIO node/edge/generated contexts were verified unchanged from the preceding full rereading; nine full native context records were read or verified against prior full readings. The documented LARC clipping rule uses a minimum of global and norm-ratio rates, unlike plain LARS multiplication. The named historical wrapper also supports a scaling mode, so the record explicitly curates the clipped identity and distinguishes wrapper modes. This control rule differs from CLARS individual-gradient normalization, LAMB final block normalization, NovoGrad scalar moment normalization and coordinate-wise Adam/RMSprop.

Claims & Grounded Evidence

7 Claims

Mechanism #clipped-rate-rule

For vanilla SGD on a block with nonzero gradient L2 norm, the documented clipping rule is lambda_k=min(gamma,eta*norm(w_k)/norm(g_k)), followed by w_k_new=w_k-lambda_k*g_k. Here gamma>=0 is the global rate and 0<=eta<1 is the trust coefficient. For this rule the fresh displacement norm is min(gamma*norm(g_k),eta*norm(w_k)). The cap is on the effective local rate; the trust coefficient is a chosen parameter, not a solved trust-region radius or an automatic batch-size schedule.

supports Source: NVIDIA OpenSeq2Seq optimizer method documentation · Locator: Layer-wise Adaptive Rate Control section, SGD and clipped local-rate equations; algebraic consequence

Mechanism #clipping-versus-scaling

The inspected historical Apex wrapper supports both clipping and scaling. On its nonzero-norm path, it computes a=eta*norm(w)/(norm(g)+d*norm(w)+epsilon). For a positive base rate gamma, clipping multiplies the supplied gradient direction by min(a/gamma,1), so vanilla SGD has effective rate min(a,gamma). Scaling mode instead multiplies by a, giving vanilla SGD effective rate gamma*a. These are distinct transformations; the scaling mode overlaps the LARS-style rule and is not evidence that every LARC configuration has the clipping identity.

supports Source: NVIDIA Apex historical LARC wrapper · Locator: LARC.__init__, class docstring and step, adaptive_lr and clip branch

Mechanism #decay-and-fresh-bound

In the historical wrapper nonzero-norm path, the direction becomes g+d*w before scaling, while the denominator is norm(g)+d*norm(w)+epsilon rather than norm(g+d*w). The wrapper temporarily sets the base optimizer weight-decay option to zero to avoid applying that option again. For d>=0, epsilon>=0, positive gamma and a defined denominator, the triangle inequality bounds the fresh vanilla-SGD displacement in clipping mode by eta*norm(w). This is coupled decay inside gradient preprocessing, not generally independent AdamW-style shrinkage.

supports Source: NVIDIA Apex historical LARC wrapper · Locator: step, weight_decays, adaptive_lr, gradient addition/scaling and base optimizer call

Limitation #wrapped-optimizer-boundary

The Apex implementation changes supplied gradients before invoking the wrapped optimizer. The vanilla-SGD local-rate interpretation and fresh-step norm bound do not automatically apply to the final displacement under accumulated momentum, adaptive coordinate moments or other stateful transformations. Momentum can retain an earlier contribution, and in the ideal first Adam step a positive common gradient multiplier can cancel in moment normalization. The wrapper therefore does not implement a universal parameter-relative displacement cap for every optimizer accepted by its interface.

supports Source: NVIDIA Apex historical LARC wrapper · Locator: Class docstring, Adam wrapper example and step calling self.optim.step after gradient rescaling

Limitation #zero-norm-and-grouping-policies

The historical wrapper skips adaptive scaling and its decay addition when either parameter norm or gradient norm is zero, and skips parameters whose gradient is None. Because base weight decay is disabled for the call, a zero-gradient parameter does not receive the wrapper decay contribution on that branch; existing optimizer state may still move it. A zero-weight block with nonzero gradient also passes an unscaled gradient to the base optimizer, differing from the zero rate of the ideal clipped formula. Blocks are parameter tensors in this implementation, not necessarily architectural layers. These are explicit versioned policies, not universal consequences of the LARC name.

supports Source: NVIDIA Apex historical LARC wrapper · Locator: step, per-parameter loop, p.grad None and nonzero-norm condition

Limitation #configuration-and-source-version

The cited Apex class delegates state_dict and load_state_dict to its wrapped optimizer; its trust coefficient, clipping flag and epsilon are not separately added by those methods. Reproducing this wrapper therefore requires retaining those settings alongside the underlying optimizer state. The selected implementation is a pinned April 2020 historical revision, while the method documentation is pinned separately to September 2020. Static reading of these sources does not establish current framework availability, dependency compatibility, exception behavior or training equivalence.

supports Source: NVIDIA Apex historical LARC wrapper · Locator: LARC.__init__, state_dict and load_state_dict
supports Source: NVIDIA OpenSeq2Seq optimizer method documentation · Locator: Pinned method documentation and configuration example

Scope #intended-use-and-evidence-scope

The documentation motivates LARC for controlling large steps in neural-network training, and the wrapper describes large-batch optimization. Those motivations and formula-level bounds do not establish universal convergence, eliminate all initialization sensitivity, prove performance preservation at arbitrary batch size, or replace a specified optimizer and schedule. This record describes the clipping rule and explicit historical wrapper boundaries without importing a numerical benchmark or claiming framework execution.

supports Source: NVIDIA OpenSeq2Seq optimizer method documentation · Locator: Layer-wise Adaptive Rate Control motivation and Notes
supports Source: NVIDIA Apex historical LARC wrapper · Locator: LARC class docstring

⚠️ Curated Limitations & Negative Results

The identity here is the documented clipping form. The named historical wrapper also supports scaling mode; clipping and global-rate multiplication must not be conflated.
The parameter-relative norm bound concerns the fresh vanilla-SGD increment with the stated signs and nonzero/defined denominators, not every wrapped optimizer final displacement or a convergence guarantee.
Zero norms, missing gradients, tensor grouping and coupled decay have explicit historical implementation policies that differ from a direct extension of the ideal equation.
The historical wrapper configuration is not fully represented by its delegated optimizer state serialization; trust coefficient, clip mode and epsilon must be retained separately.
Both pinned NVIDIA method documentation and complete historical Apex wrapper were inspected statically. No PyTorch execution, dependency audit, mixed-precision experiment or current installation compatibility is claimed.
No empirical training speed, accuracy, stability advantage or universal batch-size scaling rule is imported; constructed examples demonstrate algebraic boundaries only.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:TrainingStrategy` TrainingStrategy	related_match	LARC controls parameter-update rates during machine-learning training. The pinned TrainingStrategy target names this context but has a preprocessing genus, so this mapping expresses relatedness without asserting equivalence, subclass membership or that LARC is preprocessing.

Evidence Sources & Primary Literature

2 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`nvidiadocs`	NVIDIA OpenSeq2Seq optimizer method documentation NVIDIA contributors	official documentation documentation	ID: NVIDIA/OpenSeq2Seq@8681d381ed404fde516e2c1b823de5a213c59aba:docs/sources/source/optimizers.rst sha256:cc6d9da92b70…
`apexlarc`	NVIDIA Apex historical LARC wrapper NVIDIA contributors	official documentation documentation	ID: NVIDIA/apex@2ec84ebdca59278eaf15e8ddf32476d9d6d8b904:apex/parallel/LARC.py sha256:3d5c9836e4a7…

Curation Provenance

Curator: Codex

Method: ai_assisted

Model: GPT-6

Created: 2026-10-13T03:44:27Z

Updated: 2026-10-13T03:44:27Z

Source File: data/curated/LayerWiseAdaptiveRateControl.yaml