Home / Curated / Algorithms / Gradient Centralization

Gradient Centralization

Algorithms AIMech:GradientCentralizationOptimization DRAFT

Aliases: GC, Gradient Centralization Operator, Phi_GC, Projected Zero-Mean Gradient Centralization

A hyperparameter-free projected-gradient optimization technique for deep neural networks that operates directly on the weight gradient matrix or unfolded convolutional gradient tensor grad_W L in R^{M x N} (where M = C_in for fully connected layers and M = C_in k_1 k_2 for convolutional layers, and N = C_out) by subtracting the mean across input dimensions from each column vector grad_{w_i} L in R^M before the optimizer step: Phi_GC(grad_{w_i} L) = grad_{w_i} L - (1 / M) sum_{j=1}^M grad_{W_{j,i}} L, or in matrix form Phi_GC(grad_W L) = P grad_W L with orthogonal projection matrix P = I - e e^T (e = (1 / sqrt(M)) 1).

Identity Notes & Concept Scope Distinct from activation normalizations (Batch Normalization, Layer Normalization, Group Normalization) and from weight reparameterizations (Weight Normalization, Weight Standardization): whereas Weight Standardization enforces e^T w = 0 directly on weights and therefore breaks pretrained ImageNet checkpoints whose initial weights have non-zero mean e^T w^0 != 0, Gradient Centralization projects gradients onto the hyperplane e^T (w - w^0) = 0, preserving any pretrained initialization w^0 while embedding into SGDM, Adam, or AdamW in a single line of code.

Claims & Grounded Evidence

4 Claims

Mechanism #c-r625-hyperplane-projection-and-weight-mean-conservation

The Gradient Centralization matrix P = I - e e^T (where e = (1 / sqrt(M)) 1) satisfies P^2 = P = P^T and e^T P grad_W L = 0 (Eq. 3 and Eq. 9), so replacing grad_{w^t} L by P grad_{w^t} L performs projected gradient descent on the affine hyperplane e^T (w - w^0) = 0 (Eq. 4), holding e^T w^t = e^T w^0 exactly constant throughout training for any initialization w^0.

supports Source: Gradient Centralization: A New Optimization Technique for Deep Neural Networks · Locator: Section 4.1 Eq. (3)-(4) & Figure 2, p. 6-7; Appendix A1-A2 Eq. (9), p. 15-16

Mechanism #c-r625-output-feature-constant-intensity-shift-invariance

Theorem 4.1 (Eq. 5 and Appendix A1 Eq. 7-8) proves that when SGD or SGDM with Gradient Centralization updates a weight vector w^t = w^0 - P sum_{i=0}^{t-1} alpha^{(i)} grad_{w^{(i)}} L, the output activation difference for any input feature x and constant intensity shift x + gamma 1 satisfies (w^t)^T x - (w^t)^T (x + gamma 1) = -gamma 1^T w^0 for all t, depending only on the initial weight sum 1^T w^0 rather than the learned weights w^t.

supports Source: Gradient Centralization: A New Optimization Technique for Deep Neural Networks · Locator: Section 4.1 Theorem 4.1, Eq. (5), p. 7-8; Appendix A1 Proof of Theorem 4.1, Eq. (7)-(8), p. 15-16

Mechanism #c-r625-lipschitz-smoothing-and-gradient-explosion-suppression

Theorem 4.2 (Eq. 6 and Appendix A2 Eq. 10-11) proves that ||Phi_GC(grad_w L)||_2^2 = ||grad_w L||_2^2 - ||e^T grad_w L||_2^2 <= ||grad_w L||_2^2 and ||grad_w Phi_GC(grad_w L)||_2^2 = ||P grad_w^2 L||_2^2 = ||grad_w^2 L||_2^2 - ||e^T grad_w^2 L||_2^2 <= ||grad_w^2 L||_2^2, smoothing the optimization landscape, suppressing gradient explosion, and improving test accuracy across SGDM, Adam, Adagrad, SGDW, and AdamW on CIFAR-100, ImageNet, fine-grained classification, and COCO object detection/segmentation.

supports Source: Gradient Centralization: A New Optimization Technique for Deep Neural Networks · Locator: Section 4.2 Theorem 4.2, Eq. (6) & Figure 4, p. 8-9; Section 5.2-5.6 Tables 1-9, p. 10-15; Appendix A2 Eq. (10)-(11), p. 16-17

Limitation #c-r625-feature-normalization-and-weight-decay-coupling-limitation

Gradient Centralization operates only on weight gradients and does not normalize intermediate feature activations, so deep networks still require activation normalization (such as Batch Normalization or Group Normalization) to train effectively; moreover, when L2 weight decay lambda w is added before projection as P(g^t + lambda w), the mean component of w is not decayed unless decoupled weight decay (SGDW/AdamW) is applied outside P.

supports Source: Gradient Centralization: A New Optimization Technique for Deep Neural Networks · Locator: Section 3.4, p. 5; Section 5.2-5.3, p. 11

⚠️ Curated Limitations & Negative Results

Does not replace activation normalization layers (Batch Normalization or Group Normalization), which remain necessary to normalize feature scales across layers.
Applicable only to weight matrices/tensors with input fan-in dimension M > 1 (convolutional and fully connected weight columns), not to 1D bias vectors or scalar scale parameters.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:BatchNormalizationLayer` BatchNormalization Layer	related_match	Gradient Centralization is a statistics-based zero-mean centering operator on weight gradients complementary to Batch Normalization on activations.
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	Gradient Centralization regularizes weight and output feature spaces during deep neural network optimization.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r625-primary-pdf`	Gradient Centralization: A New Optimization Technique for Deep Neural Networks Hongwei Yong, Jianqiang Huang, Xiansheng Hua, Lei Zhang	primary paper full_text	ID: https://dblp.org/rec/conf/eccv/YongHHZ20 sha256:b94013464e17…
`r625-arxiv-abs`	Gradient Centralization: A New Optimization Technique for Deep Neural Networks (arXiv abstract) Hongwei Yong, Jianqiang Huang, Xiansheng Hua, Lei Zhang	primary paper abstract_only	ID: https://arxiv.org/abs/2004.01461 sha256:07d234bea32c…
`r625-metadata-cr`	Gradient Centralization: A New Optimization Technique for Deep Neural Networks (Crossref metadata) Hongwei Yong, Jianqiang Huang, Xiansheng Hua, Lei Zhang	primary paper abstract_only	ID: https://dblp.org/rec/conf/eccv/YongHHZ20 sha256:a7beb3267f9b…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T11:48:00Z

Updated: 2026-10-19T11:48:00Z

Source File: data/curated/GradientCentralizationOptimization.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories