Madam
Algorithms AIMech:Madam DRAFT
Aliases: Multiplicative Adaptive Moments Optimizer, B-Bit Logarithmic Number System Madam Optimizer, Compositional Deep-Relative-Trust Multiplicative Weight Update
A multiplicative adaptive moments optimization algorithm for compositional neural networks that maintains an exponential moving average of squared gradients g_bar^2 <- (1 - beta) g^2 + beta g_bar^2 and updates each weight multiplicatively via W <- clamp_{sigma*}(W odot exp[-eta * sign(W) odot clamp_{eta*/eta}(g / g_bar)]), bounding the per-step relative perturbation of every weight subset while freezing initial weight signs (Dale's principle) and supporting native B-bit quantized training on a logarithmic number system ladder W = sign(W) sigma* exp(-k eta_0).
Identity Notes & Concept Scope
Distinct from additive adaptive optimizers such as Adam and RMSprop (which add -eta g / g_bar to W and can flip weight signs and violate relative perturbation bounds on small weights) and from layer-wise relative optimizers such as LARS, LAMB, and Fromage (which require global layer-wise Frobenius norm reductions). Madam guarantees ||Delta W_*||_F / ||W_*||_F = eta locally for every subset of weights W_* without layer-level synchronization.
Claims & Grounded Evidence
4 Claims
Mechanism
#madam-compositional-descent-theorem-1-and-subset-relative-invariance
Under the deep relative trust bound ||g_k(W + Delta W) - g_k(W)||_F / ||g_k(W)||_F <= prod_{l=1}^L (1 + ||Delta W_l||_F / ||W_l||_F) - 1 for an L-layer network, the multiplicative update W -> W odot (1 - eta sign(W) odot sign(g(W))) satisfies ||Delta W_*||_F / ||W_*||_F = eta for every weight subset W_* and angle(Delta W_k, -g_k(W)) = angle(|W_k|, |g_k(W)|) =: gamma_k, guaranteeing strict loss descent whenever eta < (1 + cos gamma_k)^{1/L} - 1 for all k = 1, ..., L (Theorem 1).
Scope
#madam-sign-preservation-and-b-bit-logarithmic-ladder-scope
Madam's exponential update W <- clamp_{sigma*}(W odot exp[-eta sign(W) odot clamp_{eta*/eta}(g / g_bar)]) strictly preserves the initial sign of every weight throughout training (Dale's principle), bounds per-step relative change within exp(+-eta^*), and in B-bit Madam rounds g / g_bar to multiples of eta_0 / eta so weights move directly along the 2^B discrete rungs of the logarithmic number system W = sign(W) sigma^* exp(-k eta_0) (k in {0, ..., 2^B - 1}) without maintaining a full-precision master copy.
Application
#madam-fixed-learning-rate-and-low-bit-benchmark-applications
Using a single untuned default initial learning rate eta = 0.01 across all tasks, FP32 and 12-bit logarithmic Madam train CIFAR-10 and CIFAR-100 ResNet-18 classifiers, ImageNet ResNet-50, CIFAR-10 BigGAN class-conditional GANs (achieving 19.3 FID vs 23.9 for tuned Adam and 34 for SGD), and WikiText-2 Transformers.
Limitation
#madam-frozen-sign-expressivity-and-weight-clipping-threshold-limitation
Because multiplicative updates cannot change a zero-initialized parameter or flip the initial random sign pattern of weights, biases must be initialized away from zero, frozen signs can slightly degrade top-1 accuracy on ImageNet compared to tuned SGD (28.9% vs 24.1% test error), and explicit weight clamping at +-sigma^* is required to prevent late-stage compound exponential weight growth.
⚠️ Curated Limitations & Negative Results
- Freezes the initial sign pattern of all weights (Dale's principle) and cannot update zero-initialized parameters, requiring non-zero bias initialization and causing a ~4.8% top-1 error gap vs tuned SGD on ImageNet ResNet-50.
- Requires a weight-clamping threshold hyperparameter sigma^* (typically 1 to 5 times the initialization scale) to prevent exponential weight blow-up at the end of training.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ArtificialNeuralNetwork
Artificial Neural Network
|
related_match | Madam trains deep compositional neural networks in FP32 and low-bit logarithmic number systems via multiplicative weight updates. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
r619-primary-pdf
|
Jeremy Bernstein, Jiawei Zhao, Markus Meister, Ming-Yu Liu, Anima Anandkumar, Yisong Yue
|
primary paper
full_text
|
ID: https://dblp.org/search/publ/api?q=Learning+compositional+functions+via+multiplicative+weight+updates&format=json
sha256:9745fdc04069…
|
r619-arxiv-abs
|
Jeremy Bernstein, Jiawei Zhao, Markus Meister, Ming-Yu Liu, Anima Anandkumar, Yisong Yue
|
primary paper
abstract_only
|
ID: https://arxiv.org/abs/2006.14560
sha256:586130488af6…
|
r619-metadata-cr
|
Jeremy Bernstein, Jiawei Zhao, Markus Meister, Ming-Yu Liu, Anima Anandkumar, Yisong Yue
|
primary paper
abstract_only
|
ID: https://dblp.org/search/publ/api?q=Learning+compositional+functions+via+multiplicative+weight+updates&format=json
sha256:b67d1de202b2…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T11:42:00Z
Updated: 2026-10-19T11:42:00Z
Source File:
data/curated/Madam.yaml
· Schema:
curated_concept.yaml
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories