Variational Expectation Maximization
Algorithms AIMech:VariationalExpectationMaximization DRAFT
Aliases: Variational EM, VEM (variational expectation maximization)
A latent-variable model-fitting procedure that alternates optimization of a posterior approximation within a specified restricted family and optimization of model parameters using that approximation, targeting an evidence lower bound.
Identity Notes & Concept Scope
This record denotes variational EM with point-estimated model parameters and an explicitly restricted latent-posterior family, as analyzed by Gunawardana-Byrne. It is not all variational inference, an inference-only fit with fixed model parameters, fully Bayesian parameter inference, an amortized architecture, or a particular score/pathwise gradient estimator. Mean-field factorization is one family choice rather than the definition. Exact global coordinate solves are the clean projection realization; partial numerical solves need their own progress conditions. No normalized match across all442pinned AIO labels/scoped synonyms and480native IDs/names/aliases, including GAM and VEM probes. Full-checkout searches include ignored and hidden files in worktree and main;59and61matched lines reduce to31distinct text windows, all read. Hits denote topic-model fitting steps, prior frontier leads, unrelated GAM generalized-additive-model context or programming mnemonics. GAM is the joint distribution/parameter KL-descent framework, not GEM with its exact current E-step. Variational EM fits model parameters with a restricted posterior family; BBVI is a stochastic variational-gradient procedure, NVIL specifies amortized stochastic learning, and structured factorial-HMM inference is model-specific. Melting shares its fixed-bandwidth weighted-mean update with Gaussian mean shift; its named outer heating, membership tree and persistence selection supply the distinct scope. Bare GAM/VEM aliases are qualified. Absence is bounded to this snapshot, corpus and inspected semantic neighborhood. Identity-search counts: 59 matched lines in the worktree and 61 in main; all 31 distinct matching-text windows were read, with ignored and hidden files included.
Claims & Grounded Evidence
4 Claims
Mechanism
#restricted-e-and-parameter-m
Choose a normalized tractable family q_lambda(z) on latent states compatible with observed x and a model p_theta(x,z). At fixed theta optimize F(lambda,theta)=E_q[log p_theta(x,z)-log q_lambda(z)], equivalently minimize KL(q_lambda||p_theta(z|x)). Then, holding q fixed, maximize its expected complete-data log likelihood over theta and repeat. Exact coordinate optima give the source’s restricted projection formulation; checked partial ascent steps give a generalized realization. The latent family, model support and finite expectations must be specified.
Scope
#approximation-gap
The lower bound equals log p_theta(x)-KL(q_lambda||p_theta(z|x)). Bound ascent therefore does not generally imply observed-likelihood ascent or exact posterior recovery. At a differentiable interior fixed pair with a stationary parameter M-step, the observed log-likelihood gradient equals the theta gradient of that KL gap with q fixed. A zero gap is sufficient for the usual touching-bound argument, but zero gradient of the gap can also occur when the gap is positive; stationary error is not necessarily a local error minimum.
Limitation
#inexact-posterior-can-be-stationary
The source’s broad claim that an independence approximation permits likelihood stationarity only in an exact-family degenerate case is not adopted as a general impossibility theorem. In the saved normalized counterexample, p_theta(y,z)=Bernoulli_theta(y)*r(z), fixed correlated r has probabilities(0.3,0.2,0.2,0.3), and the unique best product approximation is uniform. Its positive KL gap is independent of theta, so variational EM still attains the observed-data likelihood optimum. This qualifies the general prose without refuting the conditional GAM theorem.
Limitation
#solver-and-theorem-boundaries
A mean-field fixed-point equation alone does not establish the global restricted E minimizer. The GAM convergence assumptions, including compactness, continuity, closed maps and sufficient strict progress, remain necessary for the cited conditional result. The printed fixed-q Boltzmann KL derivative has the opposite sign: the correct derivative is model moment minus q moment, although setting either sign to zero gives the same moment equation. Marginal independence of one pair does not alone imply a zero direct interaction when other latent paths can cancel its marginal association.
⚠️ Curated Limitations & Negative Results
- A restricted variational optimum is not automatically a stationary point of observed likelihood, a local maximum, a global optimum or an accurate posterior. Conversely, approximation error alone does not preclude likelihood stationarity.
- The broad family accommodates structured approximations; it does not equate the existing factorial-HMM procedure, BBVI, NVIL or topic models. No original Jordan1999 chapter or uninspected implementation is claimed read.
- No historical software, benchmark or source experiment was reproduced. Saved normalized finite calculations challenge bounded formula/prose claims, not the complete conditional convergence theorems.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:MachineLearningTask
Machine Learning Task
|
related_match | This procedure fits model parameters from observations; the inspected AIO target denotes a learning process/task. The algorithm-to-task relationship asserts neither equivalence nor an is-a relation. |
Evidence Sources & Primary Literature
1 Source| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
gam-source
|
Asela Gunawardana, William Byrne
|
primary paper
full_text
|
ID: https://www.jmlr.org/papers/v6/gunawardana05a.html
sha256:39339eb274d5…
|
Curation Provenance
Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-12T05:01:54Z
Updated: 2026-10-12T05:01:54Z
Source File:
data/curated/VariationalExpectationMaximization.yaml