Non-conjugate variational message passing
Algorithms AIMech:NonconjugateVariationalMessagePassing DRAFT
Aliases: NCVMP
An approximate inference algorithm that restricts variational blocks to exponential families and updates their natural parameters by matching expected-log-factor gradients through the sufficient-statistic covariance.
Identity Notes & Concept Scope
No normalized match across 442 pinned AIO IDs, names and scoped synonyms and 550 maintained IDs, names and aliases, including NCVMP and spelling variants. Full-text fixed-string searches of the recovery, older discovery and original Documents checkouts included ignored and hidden files: 34, 13 and 15 matched lines. All 26 distinct contents were read; matches are prior nonconjugate-model context, the Tan neighborhood record and its fitting leads, and unrelated programming-language tokens. Assessed all 15 semantic AIO rows and 121 native names/definitions, six complete annotated AIO CLASS nodes with incident edges and generated YAML, and six complete native neighboring records. NCVMP specifies expected-log-factor gradient matching and a Fisher-preconditioned fixed-point update, beyond choosing a mean-field family. It is distinct from BBVI score-sampling, natural evolution strategies, model-parameter VEM and query-local neighborhood selection. Wand’s full-Gaussian rewrite is represented as a specialization of this algorithm, not a duplicate new identity. The mapping and absence assessment are bounded to the inspected pin and corpus.
Claims & Grounded Evidence
7 Claims
Scope
#restricted-family
For a fixed normalizable target p(x) proportional to a product of factors f_a(x_a), choose q(x)=product_i q_i(x_i), allowing each block x_i to be vector-valued. Each q_i is a specified regular minimal exponential family q_i=exp(theta_i^T*u_i-kappa_i(theta_i)) relative to a fixed reference measure. The factorization of p must use the same product reference measure; nonconstant carrier terms cannot simply be discarded. Approximate inference minimizes KL(q||p), equivalently maximizing the expected log unnormalized target plus entropy relative to that measure. This is an inference procedure, not a requirement for neural amortization or model-parameter learning.
Mechanism
#gradient-messages
At the current proper q, hold all other blocks fixed and define S_a(theta_i)=E_q[log f_a] for factors adjacent to block i. Let C_i=Cov_q_i[u_i]=Hessian kappa_i, assumed finite and invertible. Compute each message’s natural-parameter contribution phi_ai=C_i^(-1)*grad_theta_i S_a and propose theta_i_new=sum_a phi_ai. Repeat block updates using current neighboring factors. The message contributions are potentials; individual messages need not each normalize as a standalone distribution, whereas their resulting q_i must be proper. The rule requires tractable or explicitly approximated expectations and their derivatives.
Mechanism
#stationarity-not-convergence
Where the fixed reference measure, finite expectations, differentiation and interior regularity assumptions apply, grad_theta_i KL=C_i*theta_i-sum_a grad_theta_i S_a. Thus the proposed increment theta_i_new-theta_i equals the Fisher natural-gradient direction of the variational lower bound. A valid fixed point has zero block gradients and conversely, but this algebra does not certify a local minimum, global optimum, exact posterior, monotonic bound improvement or convergence of unit steps. Approximate expectation bounds or numerical derivatives can instead yield stationary points of their approximation.
Mechanism
#conjugate-reduction
If E_q_-i[log f_a] is beta_ai^T*u_i(x_i) plus a term constant in x_i, then grad_theta_i S_a=C_i*beta_ai and the message reduces to the usual conjugate variational message beta_ai. This reduces to the conjugate message rule within the chosen factorization; it does not remove the posterior approximation caused by independence restrictions.
Limitation
#damping-and-domain
A damped block proposal mixes natural parameters as theta_i_next=(1-rho)*theta_i+rho*theta_i_new with 0<rho<=1, corresponding to multiplying old and proposed message potentials to the complementary powers. For a fixed positive rho, valid fixed points are unchanged. If both endpoints lie in the regular natural-parameter domain, convex mixing stays there; an invalid endpoint requires an explicit domain check. Damping may help but supplies no general ascent or convergence guarantee. Initialize with proper variational factors rather than assuming theta_i=0 is a normalizable uniform density on an unbounded domain.
Mechanism
#full-gaussian-specialization
For a vector block q_i=N(mu,Sigma), set S=E_q[log p_tilde(x)] with other blocks fixed, and define g=grad_mu S and symmetric G by the differential dS=g^T*dmu+tr(G^T*dSigma). Evaluate both derivatives at the current mu and Sigma. The full-Gaussian NCVMP proposal is Sigma_new=(-2*G)^(-1), mu_new=mu+Sigma_new*g, provided -2*G is positive definite. This is an algebraic specialization of the sufficient-statistic update, not a new objective or a general guarantee that the proposed covariance is valid. Correlations inside this block are retained; dependence between separately factorized blocks is omitted.
Scope
#gradient-versus-moment-matching
The local message approximation matches derivatives of expected log factors with derivatives of their exponential-family surrogate at the current variational parameters. The supplement also derives this through a fixed-cavity message update. Its analogy with expectation propagation does not identify the two algorithms: it does not replace the update by matching ordinary sufficient-statistic moments of an EP tilted distribution. Gradient-matching equations transform consistently under a differentiable change of coordinates with nonsingular Jacobian; this is not arbitrary finite-step parameterization invariance.
⚠️ Curated Limitations & Negative Results
- Regular minimal families, valid support, finite sufficient-statistic covariance and differentiable expectations are substantive assumptions. A nonminimal representation can have singular Fisher covariance; a formally written inverse is not a numerical recipe. Proper initialization and updated-factor normalization are required even if message potentials are individually improper.
- The unit proposal can leave the natural-parameter domain even for a proper target. The saved exact Gaussian example uses p(x) proportional to exp(2*x^2-x^4/4) and current q=N(0,1); the proposed precision is -1. A quarter natural-parameter step is proper in that example, without asserting bound ascent. These are curator arithmetic examples, not fitted posteriors or reproduced source experiments.
- Mean-field partition and exponential-family restrictions can produce inaccurate uncertainty or miss posterior structure. Fixed-point stationarity is weaker than convergence or optimality. Gaussian moment-coordinate simplification preserves the same proposal where well-defined; it does not make the general nonconjugate problem an exact coordinate solve.
- The main paper’s final proof line on PDF p. 3 has an entropy-derivative sign inconsistency corrected by Equation 6 on p. 4. Its zero-natural-parameter uniform initialization is not generally proper. The supplement’s parameterization argument needs a nonsingular Jacobian in addition to one-to-one labeling. The Gaussian rewrite follows Wand’s displayed derivation rather than a prose line that appears to differentiate the scalar expected-log-joint symbol prematurely.
- Logistic/softmax quadrature and auxiliary bounds are application-specific treatments, not defining requirements of general NCVMP. Source factor-of-two, sign and curvature-order issues in those variants remain recorded in the discovery notes; no separate bound algorithm is certified here. Tan spectral-fitting step-size adaptations likewise require a separate implementation and boundary audit.
- All three primary documents were read in full, including experimental contexts, but no quantitative accuracy or runtime superiority is asserted. No author implementation, fitting pipeline, training or benchmark was executed. The record remains DRAFT and author self-review is not independent approval.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:ProbabilisticGraphicalModel
Probabilistic Graphical Model
|
related_match | The primary construction uses a probabilistic factor graph and its factor-to-variable neighborhoods to organize approximate inference. The pinned AIO class denotes a graphical-model task, whereas this record denotes an inference algorithm. This is a method-to-modeling-context relation without equivalence or an is-a assertion. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
knowles2011
|
David A. Knowles, Thomas P. Minka
|
primary paper
full_text
|
ID: https://papers.nips.cc/paper_files/paper/2011/hash/5c936263f3428a40227908d5a3847c0b-Abstract.html
sha256:7c979ceb97fa…
|
wand2014
|
Matt P. Wand
|
primary paper
full_text
|
ID: https://www.jmlr.org/papers/v15/wand14a.html
sha256:e2938b069d80…
|
knowles2011-supp
|
David A. Knowles, Thomas P. Minka
|
primary paper
full_text
|
ID: https://cs.stanford.edu/people/davidknowles/knowles2011ncvmp_supp.pdf
sha256:e48b0a0caeee…
|
Curation Provenance
Curator: Codex
Method: ai_assisted
Model: GPT-6
Created: 2026-10-12T22:52:50Z
Updated: 2026-10-12T22:59:48Z