Home / Curated / Algorithms / NovoGrad

NovoGrad

Algorithms AIMech:NovoGrad DRAFT

A stochastic gradient optimizer that rescales each parameter block gradient by a scalar running second moment of that block gradient norm before accumulating momentum, with an optional weight-decay contribution added after gradient normalization.

Identity Notes & Concept Scope Compared all 442 pinned AIO and 566 native IDs, names, aliases and scoped synonyms; no normalized identity match. Ignored- and hidden-inclusive fixed-string searches of the recovery, older discovery and original Documents checkouts returned 21, 0, 0 lines. All matching lines were read and identify the pending NovoGrad lead or comparisons in the LAMB record, report, review and rendered pages, not a separate existing record. An expanded semantic search covered ten AIO rows and 90 native definitions: the preceding ten AIO and 88 native rows were verified unchanged, and the two additional AMSGrad and LAMB definitions were read. Three complete AIO node/edge/generated contexts were verified unchanged; eight full native context records were read or verified against prior full readings. NovoGrad uses scalar block second moments to normalize each fresh gradient before momentum; Adam/RMSprop use coordinate scales, LAMB normalizes a moment-scaled combined update, and LARS/CLARS use different gradient norm rules. A running-maximum NovoGrad variant is not identical to coordinate-wise AMSGrad. Shared layer-wise adaptive moments wording in the earlier paper title is not a LAMB alias.

Claims & Grounded Evidence

7 Claims

Mechanism #scalar-second-moment-before-momentum

In the published SGD-style convention, each layer or chosen parameter block has a scalar v_t=beta2*v_(t-1)+(1-beta2)*norm(g_t)^2 for the raw minibatch gradient, where norm is L2. After initialization, q_t=g_t/(sqrt(v_t)+epsilon) and m_t=beta1*m_(t-1)+q_t+d*w_t, followed by w_(t+1)=w_t-alpha_t*m_t. Here epsilon>0, alpha_t is the global rate, d is weight decay and 0<=beta1,beta2<1. Normalization occurs before momentum accumulation; normalizing an already accumulated raw-gradient momentum generally differs. This uses one second-moment scalar per block, not an Adam coordinate-wise second moment or a LAMB parameter-norm trust ratio.

supports Source: Training Deep Networks with Stochastic Gradient Normalized by Layerwise Adaptive Second Moments · Locator: Section 3, Algorithm 1, Equations 4 and 6, and parameter update, PDF pp. 2–3

Mechanism #decay-and-averaging-conventions

In the stated basic recurrence, d*w is excluded from the raw-gradient second-moment estimate and its normalization, then included in momentum. Calling this decay decoupled does not mean that it is algebraically identical to independent parameter shrinkage outside momentum. The paper also permits that alternative and an Adam-style variant multiplying the normalized-gradient-plus-decay contribution by (1-beta1). Those choices require matching initialization and rate conventions; they are not interchangeable under an arbitrary common schedule.

supports Source: Training Deep Networks with Stochastic Gradient Normalized by Layerwise Adaptive Second Moments · Locator: Section 3 Equation 6 and footnote 2, PDF p. 2; first-moment averaging variant, p. 3

Limitation #initialization-and-source-notation

The paper initializes the scalar second moment from the first squared gradient norm and gives a corresponding normalized first-gradient momentum, rather than ordinary zero-start EMA bias correction. Its initialization formulas omit epsilon, so an all-zero first gradient is undefined as printed. Algorithm 1 also leaves weight/time indexing details implicit, and Equation 5 prints a minus-weight term that conflicts with Equation 6 and the algorithm. This record follows the explicit plus-d*w rule of Equation 6 and requires a specified initialization and zero-gradient policy instead of treating every printed formula as consistent.

supports Source: Training Deep Networks with Stochastic Gradient Normalized by Layerwise Adaptive Second Moments · Locator: Algorithm 1 and Equations 5–6, PDF p. 2; initialization formulas, p. 3

Limitation #normalization-boundaries

For a subsequent step with nonnegative previous scalar state, 0<=beta2<1 and epsilon>=0 on a defined denominator, the fresh normalized-gradient norm is at most 1/sqrt(1-beta2). This bound does not constrain accumulated momentum or the complete decay-containing update to that same norm, and is not a parameter-relative trust bound or a convergence theorem. With beta2=0, unit gradient normalization requires the ideal epsilon=0 nonzero-gradient case; arbitrary momentum and decay still remain. Positive rescaling of an entire prescribed gradient history cancels in the ideal zero-epsilon normalization with consistently scaled second-moment initialization, but positive epsilon and other implementation choices can change that invariance.

supports Source: Training Deep Networks with Stochastic Gradient Normalized by Layerwise Adaptive Second Moments · Locator: Equations 4 and 6, PDF p. 2, and Notes 1–2, p. 3; algebraic consequences under the stated conditions

Limitation #implementation-rate-and-zero-policy

The inspected TensorFlow Addons dense path initializes v from the squared gradient norm at global optimizer iteration zero, uses sqrt(v)+epsilon even then, adds decay after normalization, and optionally averages the contribution after the first iteration. It calls ResourceApplyKerasMomentum with Nesterov disabled. The inspected TensorFlow API defines its velocity recurrence as a_t=beta1*a_(t-1)-alpha_t*q_t followed by w+=a_t, with q_t here denoting the complete supplied contribution. This rate-inside-velocity convention can differ from the paper current-rate-outside-momentum convention when the schedule changes. Parameter tensors define the implementation blocks; sparse behavior, exclusions, helper dependencies and runtime require separate assessment.

supports Source: TensorFlow Addons NovoGrad optimizer implementation · Locator: _create_slots, _resource_apply_dense and _resource_apply_sparse; full file read statically
supports Source: TensorFlow ResourceApplyKerasMomentum API reference · Locator: API signature and displayed momentum recurrence, use_nesterov=False

Limitation #state-memory-and-running-max

For N scalar parameters partitioned into L blocks, the basic mathematical state uses N first-moment entries and L scalar second moments, compared with Adam two N-sized moment arrays. At equal precision this is (N+L)/(2N) of that moment storage, approximately half only when L is much smaller than N. It is not a claim about total training memory or every library variant. The paper proposes a running-maximum second-moment variant; a nonincreasing effective rate additionally requires a suitable global rate schedule, and this is not an unconditional convergence guarantee. The basic method is discussed as admitting stochastic convex counterexamples.

supports Source: Training Deep Networks with Stochastic Gradient Normalized by Layerwise Adaptive Second Moments · Locator: Section 2.3, PDF p. 2; running-maximum discussion and Notes 3, p. 3

Scope #applications-and-evaluation-scope

The paper evaluates selected image classification, speech recognition, machine translation and language-modeling configurations. Schedules, decay, dropout, hardware and accumulation settings vary, and some configurations explicitly use warm-up. These experiments do not establish a universal no-warm-up rule, memory reduction for a whole training system, or performance advantage across tasks. This record imports no numerical benchmark and makes no training reproduction claim.

supports Source: Training Deep Networks with Stochastic Gradient Normalized by Layerwise Adaptive Second Moments · Locator: Sections 4.1–4.6, Tables 1–7 and Figures 1–2, PDF pp. 3–5; conclusion, pp. 5–6

⚠️ Curated Limitations & Negative Results

The v3 PDF title differs from the arXiv landing title. Its Equation 5, initialization and loop indexing contain inconsistencies or omissions; the identity follows Section 3 Equation 6 with explicit convention boundaries.
Basic SGD-style momentum, the averaged variant and direct shrinkage alternatives differ unless rate and state conventions are reconciled. The later TensorFlow Addons dense implementation invokes a rate-inside-velocity kernel.
The theoretical fresh-gradient normalization bound is not a bound on full momentum, decay or parameter-relative displacement and does not establish convergence or generalization.
The approximate half-memory comparison concerns mathematical moment arrays with few blocks and equal precision, excluding parameters, gradients, activations and other training storage. Implementation variants can allocate other state.
The complete seven-page v3 preprint, complete pinned TensorFlow Addons optimizer file and the cited TensorFlow API section were inspected. The library revision postdates the paper; dependency helpers, compiled kernels and runtime were not executed or fully audited, and sparse behavior is not claimed equivalent to dense updates.
Experiments use differing configurations and include warm-up; source table/rate differences remain in the discovery notes. No benchmark values, universal superiority, independent proof certification or training result are imported.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:TrainingStrategy` TrainingStrategy	related_match	NovoGrad directly controls machine-learning parameter training, the central context named by TrainingStrategy. The pinned target has a preprocessing genus and learning-task examples, so contextual relatedness does not assert optimizer equivalence, subclass membership or that the algorithm is preprocessing.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`ginsburg2020`	Training Deep Networks with Stochastic Gradient Normalized by Layerwise Adaptive Second Moments Boris Ginsburg, Patrice Castonguay, Oleksii Hrinchuk, Oleksii Kuchaiev, Ryan Leary, Vitaly Lavrukhin, Jason Li, Huyen Nguyen, Yang Zhang, Jonathan M. Cohen	primary paper full_text	ID: arXiv:1905.11286v3 sha256:6c554ab644b3…
`tfaddons`	TensorFlow Addons NovoGrad optimizer implementation The TensorFlow Authors	official documentation documentation	ID: tensorflow/addons@d208d752e98c310280938efa939117bf635a60a8:tensorflow_addons/optimizers/novograd.py sha256:a8852c9d2541…
`tfmomentum`	TensorFlow ResourceApplyKerasMomentum API reference The TensorFlow Authors	official documentation documentation	sha256:3dc5cef573ef…

Curation Provenance

Curator: Codex

Method: ai_assisted

Model: GPT-6

Created: 2026-10-13T03:28:02Z

Updated: 2026-10-13T03:28:02Z

Source File: data/curated/NovoGrad.yaml