Home / Curated / Algorithms / Fromage (Frobenius Matched Gradient Descent)

Fromage (Frobenius Matched Gradient Descent)

Algorithms AIMech:FromageFrobeniusMatchedGradientDescentOptimization DRAFT

Aliases: Fromage Optimizer, Frobenius Matched Gradient Descent, Deep Relative Trust Gradient Descent (Fromage)

A layerwise relative-trust optimization algorithm for deep neural networks (Algorithm 1) derived from the deep relative trust gradient breakdown model and neural network descent lemma (Lemma 2) that updates each layer's weight matrix W_l (l = 1, ..., L) with gradient g_l = grad_{W_l} L(W) via W_l <- (1 / sqrt(1 + eta^2)) [W_l - eta (||W_l||_F / ||g_l||_F) g_l], inducing a fixed relative Frobenius perturbation ||Delta W_l||_F / ||W_l||_F = eta per step while preventing compounding norm growth in scale-invariant layers.

Identity Notes & Concept Scope Distinct from LARS (Round 594) and LAMB (Round 593) by deriving layerwise relative step scaling from a multiplicative compositional perturbation bound on multilayer perceptrons and their layer-to-output Jacobians (Theorem 1 & Modelling Assumption 1: Deep Relative Trust) together with the exact Pythagorean normalization prefactor 1 / sqrt(1 + eta^2), eliminating weight-decay hyperparameter tuning for rescaling-invariant layers.

Claims & Grounded Evidence

4 Claims

Mechanism #c-r599-compositional-perturbation-bound-and-deep-relative-trust-lemma

For an L-layer multilayer perceptron with (alpha, beta)-transmitting nonlinearity and matrices of condition number at most kappa, Theorem 1 bounds relative functional change by ((beta / alpha) kappa^2)^L [prod_{k=1}^L (1 + ||Delta W_k||_F / ||W_k||_F) - 1] and relative Jacobian change analogously (Section 3 & Appendix B, pp. 4, 14-17), which under Deep Relative Trust guarantees loss descent for layerwise perturbations ||Delta W_l||_F / ||W_l||_F <= eta whenever eta < (1 + cos theta_l)^{1/L} - 1 (Lemma 2, pp. 6, 13).

supports Source: On the distance between two neural networks and the stability of learning · Locator: Section 3-4, Theorem 1, Lemma 1-2, & Modelling Assumption 1, pp. 4-6; Appendix A-B, pp. 13-17

Mechanism #c-r599-pythagorean-prefactor-stability-on-scale-invariant-layers

In batch-normalized or spectrally normalized layers that are invariant to W_l -> (1 + alpha) W_l, the gradient g_l is orthogonal to W_l, so unregularized LARS updates Delta W_l = -eta (||W_l||_F / ||g_l||_F) g_l compound weight norms by ||W_l + Delta W_l||_F^2 = (1 + eta^2) ||W_l||_F^2 at every step; Fromage's prefactor 1 / sqrt(1 + eta^2) exactly cancels this Pythagorean growth without a weight-decay hyperparameter (Section 5 & Figure 2, p. 7).

supports Source: On the distance between two neural networks and the stability of learning · Locator: Section 5, Eq. (5)-(6), Algorithm 1, & Figure 2, p. 7

Application #c-r599-deep-mlp-and-single-lr-benchmark-training

Fromage trains vanilla ReLU multilayer perceptrons (without batch normalization or skip connections) up to depth L = 50 on MNIST where SGD and Adam fail beyond depth 25 (Figure 3, p. 8), and using a single untuned default learning rate eta = 0.01 across CIFAR-10 ResNet-18, ImageNet ResNet-50, CIFAR-10 cGAN (FID 16 +/- 1 vs Adam 23.7 +/- 0.7), and WikiText-2 Transformer matches or outperforms tuned SGD and Adam baselines (Section 6, Table 1-2, pp. 8-9, 18).

supports Source: On the distance between two neural networks and the stability of learning · Locator: Section 6, Figures 3-4 & Table 1, pp. 8-9; Appendix C & Table 2, pp. 18-19

Limitation #c-r599-depth-dependent-step-size-and-norm-cap-regularization

On very deep skip-free multilayer perceptrons, Lemma 2 and Figure 3 require decreasing eta as depth L grows (since (1 + cos theta_l)^{1/L} - 1 = O(1 / L)), and on CIFAR-10 ResNet-18 and WikiText-2 Transformer benchmarks Fromage heavily overfits the training set unless each layer's Frobenius norm is constrained not to exceed its initial norm.

supports Source: On the distance between two neural networks and the stability of learning · Locator: Section 6-7, Figure 3, pp. 8-9; Appendix C, p. 19

⚠️ Curated Limitations & Negative Results

Deep Relative Trust is a modelling assumption that sets nonlinearity transmission parameters alpha = beta = 1/2, matrix condition numbers kappa = 1, and omits the output loss derivative dL/df.
Without bounding layer Frobenius norms by their initial values, Fromage can severely overfit training sets on ResNet-18 CIFAR-10 and WikiText-2 Transformer tasks.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:ArtificialNeuralNetwork` Artificial Neural Network	related_match	Fromage is a layerwise relative-trust gradient descent algorithm derived from compositional perturbation bounds on deep artificial neural networks.
`AIO:BatchNormalizationLayer` BatchNormalization Layer	related_match	Fromage's 1 / sqrt(1 + eta^2) prefactor specifically corrects the compounding weight-norm growth caused by orthogonal gradients in batch-normalized and spectrally normalized neural network layers.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r599-primary-pdf`	On the distance between two neural networks and the stability of learning Jeremy Bernstein, Arash Vahdat, Yisong Yue, Ming-Yu Liu	primary paper full_text	ID: https://dblp.org/rec/conf/nips/BernsteinVY020 sha256:459ff2910497…
`r599-arxiv-abs`	On the distance between two neural networks and the stability of learning (arXiv abstract) Jeremy Bernstein, Arash Vahdat, Yisong Yue, Ming-Yu Liu	primary paper abstract_only	ID: https://arxiv.org/abs/2002.03432 sha256:d3ee155a3dd4…
`r599-metadata-cr`	On the distance between two neural networks and the stability of learning (Crossref metadata) Jeremy Bernstein, Arash Vahdat, Yisong Yue, Ming-Yu Liu	primary paper abstract_only	ID: https://dblp.org/rec/conf/nips/BernsteinVY020 sha256:e4161fbb7904…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T11:22:00Z

Updated: 2026-10-19T11:22:00Z

Source File: data/curated/FromageFrobeniusMatchedGradientDescentOptimization.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories