Home / Curated / Algorithms / FOOF (Fast First-Order Optimizer / First-Order Gradient Descent on Neurons)

FOOF (Fast First-Order Optimizer / First-Order Gradient Descent on Neurons)

Algorithms AIMech:FoofFirstOrderOnFeaturesNeuronGradientDescentOptimization DRAFT

Aliases: FOOF, Fast First-Order Optimizer, First-Order-On-Features, Gradient Descent on Neurons

A layer-wise neural network optimization algorithm that performs gradient descent in neuron activation space rather than weight space by solving a ridge-regularized least-squares problem for the weight update Delta_W that moves layer pre-activations B = W A along their loss gradient E = dL/dB, resulting in the update (Delta_W)^T = eta * (lambda * I + A A^T)^{-1} A E^T that preconditions standard weight gradients solely by the inverse damped input-activation covariance (lambda * I + A A^T)^{-1} without any second-order loss or output-gradient curvature factor.

Identity Notes & Concept Scope Distinct from K-FAC (AIMech:KfacKroneckerFactoredApproximateCurvature), which preconditions weight gradients from both sides using both the input-activation covariance factor (A A^T + lambda_A I)^{-1} and the Monte-Carlo Fisher output-gradient factor (E_F E_F^T + lambda_E I)^{-1}; FOOF completely omits the second Kronecker factor (and its extra Fisher backward pass), relying strictly on zero-th order input activations A A^T and first-order gradients A E^T. Distinct from Natural Neural Networks (Desjardins et al., 2015), which reparameterize layers via explicit singular value decompositions of centered activations rather than amortizing direct damped matrix inversions (Sigma_l + lambda I)^{-1}.

Claims & Grounded Evidence

5 Claims

Mechanism #foof-neuron-space-proximal-least-squares-update-mechanism

For a layer with weight matrix W in R^{n x m}, input activations A in R^{m x D}, pre-activations B = W A in R^{n x D}, and backpropagated output derivatives E = dL/dB in R^{n x D}, FOOF minimizes min_{Delta_W} || (Delta_W) A - eta * E ||_F^2 + (lambda / 2) || Delta_W ||_F^2 to obtain the closed-form update (Delta_W)^T = eta * (lambda * I + A A^T)^{-1} A E^T, resolving conflicting per-datapoint gradients in neuron output space.

supports Source: Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization · Locator: Section 4 Equations (5)-(6) and Appendix F Toy Example, pages 5-6, 19

Mechanism #foof-amortized-input-covariance-inversion-mechanism

In stochastic mini-batch training (Algorithm 1), FOOF maintains an exponential moving average Sigma_l <- m * Sigma_l + (1 - m) A_l A_l^T of the input-activation second moment during the S steps preceding every T-th step, and recomputes the damped inverse P_l = (Sigma_l + lambda I)^{-1} only once every T steps, eliminating K-FAC's extra Fisher backward pass and second matrix inversion.

supports Source: Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization · Locator: Section 4.1, Appendix D.1, and Appendix K Algorithm 1, pages 6, 15, 33

Scope #foof-heuristic-damping-kfac-alignment-scope

Controlled ablations against exact subsampled and full-batch natural gradients demonstrate that K-FAC with theoretically standard damping (A A^T (x) E_F E_F^T + lambda I) performs no better than SGD, whereas factored heuristic damping (A A^T + lambda_A I) (x) (E_F E_F^T + lambda_E I) drives the second Kronecker factor toward a scalar multiple of the identity and increases cosine similarity between K-FAC and FOOF toward 1.

supports Source: Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization · Locator: Section 3.1-3.2 Figures 1-3 and Section 4.2 Figures 3B-4C, pages 2-7

Application #foof-mlp-and-wideresnet-speedup-application

Across MLPs on Fashion-MNIST and MNIST, VGG11 on SVHN, and Wide ResNet-18 on CIFAR-10 and CIFAR-100, FOOF achieves lower or comparable training loss and faster per-update progress than K-FAC, Adam, and SGD with momentum while running approximately 1.5x faster than K-FAC in wall-clock time.

supports Source: Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization · Locator: Section 4.2 Figures 4-5 and Appendix J.5 Figures J.16-J.18, pages 6-7, 30

Limitation #foof-deep-autoencoder-trajectory-limitation

On deep bottleneck autoencoder benchmarks (MNIST, CURVES, and FACES), although a single FOOF update along K-FAC's trajectory achieves greater immediate mini-batch loss reduction than K-FAC, training along FOOF's own full trajectory reaches slightly higher final reconstruction loss than K-FAC.

supports Source: Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization · Locator: Section 5 Limitations and Appendix J.1 Figures J.19-J.21, pages 7-8, 26, 31

⚠️ Curated Limitations & Negative Results

Requires storing and periodically inverting an m x m input-activation covariance matrix (Sigma_l + lambda I) for each layer (costing O(m^3) every T steps), and extracting unfolded patches for convolutional layers.
Slightly underperforms K-FAC on full-trajectory deep autoencoder reconstruction benchmarks despite making larger single-step local loss reductions along K-FAC's trajectory.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	FOOF is a layer-wise neuron-space optimization algorithm for training feedforward and convolutional neural networks.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r643-primary-pdf`	Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization Frederik Benzing	primary paper full_text	ID: https://dblp.org/rec/conf/icml/Benzing22 sha256:1053cdde369d…
`r643-arxiv-abs`	Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization (arXiv abstract) Frederik Benzing	primary paper abstract_only	ID: https://arxiv.org/abs/2201.12250 sha256:10fcecb430ab…
`r643-metadata-cr`	Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization (Crossref metadata) Frederik Benzing	primary paper abstract_only	ID: https://dblp.org/rec/conf/icml/Benzing22 sha256:ced043708213…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T12:06:00Z

Updated: 2026-10-19T12:06:00Z

Source File: data/curated/FoofFirstOrderOnFeaturesNeuronGradientDescentOptimization.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories