Improved Variational Online Newton (IVON)
Algorithms AIMech:IvonImprovedVariationalOnlineNewtonForLargeDeepNetworks DRAFT
Aliases: IVON, Improved Variational Online Newton, IVON Variational Deep Learning Optimizer, Reparameterized Variational Online Newton
Improved Variational Online Newton (IVON) is a scalable variational Bayesian learning algorithm for deep neural networks that optimizes a diagonal Gaussian posterior q(theta) = N(m, diag(sigma^2)) over network weights with nearly identical computational and memory cost to Adam. Rather than computing expensive per-sample squared gradients or Gauss-Newton Hessian approximations, IVON estimates the diagonal expected Hessian h strictly from standard mini-batch stochastic gradients ghat evaluated at a single weight perturbation theta ~ q(theta) via Stein's identity / the reparameterization trick: hhat = ghat * (theta - m) / sigma^2. It combines an exponential moving average of hhat with a strictly positive Riemannian natural-gradient update ensuring h + delta > 0 without coordinate clipping, and sets the effective preconditioner scale to sigma = 1 / sqrt(lambda * (h + delta)) where lambda is the effective sample size.
Identity Notes & Concept Scope
Must be distinguished from standard deterministic adaptive optimizers (Adam, AdamW, AdaHessian) which maintain point estimates m without sampling theta ~ N(m, diag(sigma^2)) and therefore neither optimize the variational free energy L(q) = lambda * E_q[ell_bar(theta)] + D_KL(q || p) nor yield posterior weight variances sigma^2 for Bayesian model averaging, weight pruning, or uncertainty calibration. Also distinct from Variational Online Newton (VON, Khan et al., 2018) and Variational Online Gauss-Newton (VOGN, Osawa et al., 2019), which require per-example gradient squares or explicit Hessian-vector products that incur prohibitive memory and runtime overhead on large language models such as GPT-2 and ResNet-50.
Claims & Grounded Evidence
4 Claims
Mechanism
#r650-claim-reparam-hessian-estimator
Under a Gaussian variational posterior q(theta) = N(m, diag(sigma^2)), Stein's identity / the reparameterization trick yields E_q[nabla^2 ell_bar(theta)] = E_q[nabla ell_bar(theta) * (theta - m) / sigma^2], allowing IVON to obtain an unbiased single-sample stochastic diagonal Hessian estimator hhat = ghat * (theta - m) / sigma^2 using only the standard mini-batch gradient ghat at theta ~ q(theta) with zero extra backpropagation passes.
Mechanism
#r650-claim-positive-riemannian-hessian-update
To prevent negative diagonal Hessian estimates from invalidating the Gaussian variance sigma^2 = 1 / (lambda * (h + delta)), IVON incorporates a higher-order Riemannian correction term in the exponential moving average update h <- (1 - beta_2) * h + beta_2 * hhat + 0.5 * (1 - beta_2)^2 * (h - hhat)^2 / (h + delta), which guarantees h_t + delta > 0 whenever h_{t-1} + delta > 0 and hhat >= -(h_{t-1} + delta).
Application
#r650-claim-gpt2-llm-and-uncertainty-applications
IVON trains large-scale foundation models from scratch—including GPT-2 (125M to 773M parameters on OpenWebText) and ResNet-50 on ImageNet—matching or improving upon AdamW's validation perplexity and accuracy while directly providing posterior weight variances for calibrated predictive uncertainty, out-of-distribution detection, sensitivity analysis, and model merging.
Limitation
#r650-claim-effective-sample-size-tuning-limitation
IVON introduces an explicit effective sample size hyperparameter lambda and initial Hessian scale h_0 that govern the exploration noise sigma = 1 / sqrt(lambda * (h + delta)); mis-specifying lambda (too small) injects excessive weight perturbation noise that degrades optimization, whereas setting lambda too large collapses q(theta) toward a deterministic point estimate.
⚠️ Curated Limitations & Negative Results
- Requires tuning the effective dataset size hyperparameter lambda and initial diagonal Hessian scale h_0 to balance weight-perturbation exploration noise against optimization stability.
- Restricted to a mean-field (diagonal) Gaussian posterior approximation q(theta) = N(m, diag(sigma^2)), omitting off-diagonal weight correlations across parameters.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ArtificialNeuralNetwork
Artificial Neural Network
|
related_match | IVON is a variational Bayesian optimization algorithm designed for training large deep artificial neural networks such as Transformers and residual convolutional networks. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
r650-primary-pdf
|
Yuesong Shen, Nico Daheim, Bai Cong, Peter Nickl, Gian Maria Marconi, Clement Bazan, Rio Yokota, Iryna Gurevych, Daniel Cremers, Mohammad Emtiyaz Khan, Thomas Möllenhoff
|
primary paper
full_text
|
ID: https://dblp.org/rec/conf/icml/ShenDCNMBYGCKM24.html
sha256:f82c661abb25…
|
r650-arxiv-abs
|
Yuesong Shen, Nico Daheim, Bai Cong, Peter Nickl, Gian Maria Marconi, Clement Bazan, Rio Yokota, Iryna Gurevych, Daniel Cremers, Mohammad Emtiyaz Khan, Thomas Möllenhoff
|
primary paper
abstract_only
|
ID: https://arxiv.org/abs/2402.17641
sha256:be56c7f0aa3c…
|
r650-metadata-cr
|
Yuesong Shen, Nico Daheim, Bai Cong, Peter Nickl, Gian Maria Marconi, Clement Bazan, Rio Yokota, Iryna Gurevych, Daniel Cremers, Mohammad Emtiyaz Khan, Thomas Möllenhoff
|
primary paper
abstract_only
|
ID: https://dblp.org/rec/conf/icml/ShenDCNMBYGCKM24.html
sha256:4e25e28d8788…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T12:13:00Z
Updated: 2026-10-19T12:13:00Z
Source File:
data/curated/IvonImprovedVariationalOnlineNewtonForLargeDeepNetworks.yaml
· Schema:
curated_concept.yaml
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories