Home / Curated / Algorithms / Cross-validated prior-likelihood conflict diagnostic

Cross-validated prior-likelihood conflict diagnostic

Algorithms AIMech:CrossValidatedPriorLikelihoodConflictDiagnostic DRAFT

A Bayesian hierarchical-model diagnostic that leaves an assessed unit out of population-parameter fitting, generates predictive-prior and reference-prior likelihood replicates of its unit effect, and measures their signed conflict under an explicitly specified replication law. Shared nuisance parameters require controlled feedback and dependence handling.

Identity Notes & Concept Scope No normalized identity match for the proposed unit-level cross-validated and general separator-node conflict diagnostics across 442 pinned AIO records and 557 maintained IDs, names, aliases and scoped synonyms. Fixed-string searches included ignored and hidden files in the recovery, older discovery and original Documents checkouts, returning 121, 0 and 0 lines. All 121 were assessed, including 19 long rows read in bounded groups after verifying three duplicate pairs. They contain prior fitting and Gaussian-message records and their mirrors, source metadata and pending comparator leads. Assessed 55 semantic AIO rows and 202 native names/definitions: 50 and 177 were verified equal to earlier complete readings, and all 5 and 25 additional rows were read. Ten complete AIO nodes, annotations, edges and generated records were verified against earlier readings; five native context records were read or verified unchanged. This procedure is defined by unit-level held-out hyperparameter fitting and prior/likelihood replication. It differs from fitting algorithms, Gaussian expected-log message screening, neighbor-density outlier scores and general arbitrary-separator evidence splitting. This is a descriptive AIMech algorithm identity, not a blanket alias for every prior-likelihood conflict method.

Claims & Grounded Evidence

7 Claims

Scope #unit-level-split

For a hierarchical model with conditionally independent units y_i|phi_i,gamma and random unit effects phi_i|beta, leave the entire assessed unit y_i out when learning the population and nuisance parameters. Fit H_i(beta,gamma)=p(beta,gamma|y_minus_i), then compare information about phi_i from this remaining-data predictive prior with information from the assessed unit likelihood and an explicit reference prior. The held-out unit can contain a vector of observations. Require a well-defined remaining-data posterior and proper replication distributions; omitting one observation from a multi-observation unit defines a different check.

supports Source: Identifying outliers in Bayesian hierarchical models: a simulation-based approach · Locator: Sections 3–4, journal pp. 415–418 / PDF pp. 7–10

Mechanism #replicate-construction

Without shared unknown nuisance parameters, draw phi_prior by first drawing beta from its remaining-data posterior and then drawing phi_prior~p(phi_i|beta). Independently draw phi_lik from the normalized density proportional to p(y_i|phi_i)*r_i(phi_i), where r_i is a stated reference prior rather than the fitted hierarchical prior being tested. The reference prior may be improper only when this local posterior is proper. A uniform prior is justified for the symmetric location construction in the source; Jeffreys priors are proposed for other likelihood parameters, with no general guarantee of posterior propriety, negligible prior influence or calibration.

supports Source: Identifying outliers in Bayesian hierarchical models: a simulation-based approach · Locator: Section 4, journal pp. 417–420 / PDF pp. 9–12; Appendix A3, journal p. 432 / PDF p. 24
supports Source: Conflict Diagnostics in Directed Acyclic Graphs, with Applications in Bayesian Evidence Synthesis · Locator: Sections 2.2 and 3.1, PDF pp. 3–4 and 7–8

Mechanism #nuisance-cut

When gamma is needed by the held-out likelihood, the source uses nuisance draws learned from the other units and prevents y_i from feeding back into them. An explicit conditional simulation construction draws (beta,gamma) from H_i, then phi_prior~p(phi_i|beta) and phi_lik from the normalized conditional local density l_i(phi|y_i,gamma) proportional to p(y_i|phi,gamma)*r_i(phi|gamma), with independent local random draws given that shared hyperparameter draw. The joint replication law and pairing must be stated: shared or correlated nuisance uncertainty can induce dependence, so Var(phi_prior-phi_lik) is not automatically the sum of marginal variances. A cut freezes the nuisance marginal against target-unit feedback; it does not by itself split all dependence. Normalizing the local density for each gamma before mixing generally differs from mixing unnormalized likelihood kernels and then normalizing, because the latter can change the nuisance weights. This normalization-order distinction is curator reasoning for the stated conditional construction.

supports Source: Identifying outliers in Bayesian hierarchical models: a simulation-based approach · Locator: Section 4 nuisance-parameter discussion, journal p. 420 / PDF p. 12
supports Source: Conflict Diagnostics in Directed Acyclic Graphs, with Applications in Bayesian Evidence Synthesis · Locator: Section 2.2 cut versus split, PDF p. 4; Section 4.3 shared residual variance, PDF p. 17
supports Source: A stochastic variational framework for fitting and diagnosing generalized linear mixed models · Locator: Appendix C conditional nuisance and likelihood-replicate simulation, PDF p. 39; compare Section 5.1 discussion, PDF p. 18

Mechanism #signed-conflict-score

For continuous scalar replicates define delta=phi_prior-phi_lik and estimate the lower-tail probability P(delta<=0) from the specified paired replication law. Its complementary upper tail is P(delta>0); a declared two-sided equal-tail score is twice the smaller tail. Tail orientation is part of the diagnostic, and small lower versus small upper scores flag opposite directions. Monte Carlo proportions need adequate effective sampling and uncertainty assessment. The two-sided equal-tail score is not a posterior probability that the model is true and is not generally a density-level-set conflict score for skewed or multimodal differences. Discrete differences require an explicit treatment of ties; a predictive mid-p convention is not a theorem of exact continuous uniformity.

supports Source: Identifying outliers in Bayesian hierarchical models: a simulation-based approach · Locator: Section 2.2, journal p. 415 / PDF p. 7; Section 4, Equation 7, journal p. 418 / PDF p. 10; discrete example, journal p. 420 / PDF p. 12
supports Source: Conflict Diagnostics in Directed Acyclic Graphs, with Applications in Bayesian Evidence Synthesis · Locator: Section 3.2.2, PDF p. 9

Mechanism #gaussian-equivalence

In the known-variance Gaussian hierarchy phi_i|beta~N(beta,Omega), y_i|phi_i~N(X_i*phi_i,sigma^2*I), require sigma^2>0, full column rank X_i and positive-definite Omega. Let the remaining-data beta posterior be N(m_minus,Gamma_minus). A flat reference prior for phi_i gives phi_lik~N(phi_hat,R_i), where phi_hat=(X_i^T*X_i)^-1*X_i^T*y_i and R_i=sigma^2*(X_i^T*X_i)^-1. The predictive-prior replicate is N(m_minus,Omega+Gamma_minus). Their independent difference is N(m_minus-phi_hat,Omega+Gamma_minus+R_i); its mean Mahalanobis statistic gives the same Gaussian reference-tail comparison as predicting phi_hat directly. Observation-estimator noise R_i enters once. More generally, the source proves equality with the mixed-predictive scalar comparison for a sufficient-statistic-based location estimator with a symmetric pivotal sampling distribution and a flat reference prior. This equivalence is conditional on those constructions, not universal for hierarchical models.

supports Source: Identifying outliers in Bayesian hierarchical models: a simulation-based approach · Locator: Section 4, Equations 8–11, journal p. 419 / PDF p. 11; Appendix A2, Equations 13–15, journal pp. 430–431 / PDF pp. 22–23; Appendix A3, journal p. 432 / PDF p. 24

Limitation #calibration-boundary

For an exact continuous predictive comparison, the probability integral transform gives uniformity under the matching conditional predictive law; integrating a proper prior gives the corresponding preposterior statement. Neither statement alone gives uniformity for every fixed population parameter under repeated sampling. Transferring predictive calibration to this conflict score requires an applicable equivalence result or separate assessment. Pivotal known-variance location constructions with appropriate flat reference priors supply a special case. An informative proper population prior, posterior normality, a chi-square reference tail, leave-one-unit-out refitting or a Jeffreys local prior alone does not establish general frequentist calibration. Estimated variances, shared nuisance parameters and non-Gaussian differences require explicit qualification.

supports Source: Identifying outliers in Bayesian hierarchical models: a simulation-based approach · Locator: Appendix A1, journal pp. 429–430 / PDF pp. 21–22; Appendix A2 known-variance setting, journal pp. 430–431 / PDF pp. 22–23
supports Source: Conflict Diagnostics in Directed Acyclic Graphs, with Applications in Bayesian Evidence Synthesis · Locator: Scope of normal-model results, PDF p. 7; discussion of calibration and sensitivity, PDF pp. 19–20

Scope #diagnostic-use-and-approximations

The procedure tests compatibility of the assumed unit-effect distribution with unit-likelihood information and points to model or data follow-up; it does not identify which source is wrong or authorize removing a unit. Repeated refits can be costly. Reusing full-data hyperparameters and the later Gaussian variational-message diagnostic are distinct approximations, not exact implementations of this leave-one-unit-out construction. Scores across assessed units can be dependent and multiple-testing corrections need their own assumptions. The source application comparisons and normal-model examples do not establish a universal detection rate, automatic exclusion threshold or clinical decision rule.

supports Source: Identifying outliers in Bayesian hierarchical models: a simulation-based approach · Locator: Sections 3 and 6–8, journal pp. 415–417 and 422–429 / PDF pp. 7–9 and 14–21
supports Source: Conflict Diagnostics in Directed Acyclic Graphs, with Applications in Bayesian Evidence Synthesis · Locator: Discussion, PDF pp. 19–20
supports Source: A stochastic variational framework for fitting and diagnosing generalized linear mixed models · Locator: Gaussian-message approximation, Sections 5.1–5.2, PDF pp. 17–21

⚠️ Curated Limitations & Negative Results

A well-defined hierarchical posterior does not ensure that a held-out fit or local reference-prior replication is proper. Reference-prior choice, parameter scale, rank and identifiability need case-specific checks.
A shared nuisance cut requires a specified joint simulation law and normalization order. It prevents a selected feedback path, not all dependence; independent marginal recombination can change the diagnostic.
The original 2007 PDF p. 11 displays a difference mean with the opposite sign to its prior-minus-likelihood definition, and Figure 4 on PDF p. 31 gives an inconsistent inequality. Appendix A2 on PDF p. 23 also refers to Equation 13, a predictive distribution for the estimator, when discussing the latent prior replicate. The record follows the declared difference and generative hierarchy, retaining observation-estimator variance only once as in Equation 15.
Known-variance Gaussian equivalence and a proper-prior preposterior calibration statement do not imply fixed-parameter uniformity for arbitrary informative priors. Tail direction, discreteness, nuisance coupling, dependence across units and Monte Carlo error remain substantive limitations.
All 36 Marshall–Spiegelhalter publisher pages, 22 Presanis et al. electronic-reprint pages and 42 Tan–Nott preprint pages were read. No author software, MCMC, model fitting or empirical calibration experiment was run. Independent analytic examples check Gaussian equivalence, fixed-parameter versus preposterior size, conditional normalization and nuisance-induced covariance; they do not reproduce source experiments.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:RandomEffectsModel` Random Effects Model	related_match	The diagnostic compares information about hierarchical random unit effects against their population distribution. RandomEffectsModel provides this defining model context but does not denote the held-out replication and conflict-scoring procedure; the mapping asserts relatedness, not equivalence or subclass membership.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`marshall2007`	Identifying outliers in Bayesian hierarchical models: a simulation-based approach E. C. Marshall, D. J. Spiegelhalter	primary paper full_text	ID: https://doi.org/10.1214/07-BA218 sha256:289815a344a2…
`presanis2013`	Conflict Diagnostics in Directed Acyclic Graphs, with Applications in Bayesian Evidence Synthesis Anne M. Presanis, David Ohlssen, David J. Spiegelhalter, Daniela De Angelis	primary paper full_text	ID: https://doi.org/10.1214/13-STS426 sha256:9cff5092b47d…
`tan2014`	A stochastic variational framework for fitting and diagnosing generalized linear mixed models Linda S. L. Tan, David J. Nott	primary paper full_text	ID: https://doi.org/10.1214/14-BA885 sha256:276ebdc67a5f…

Curation Provenance

Curator: Codex

Method: ai_assisted

Model: GPT-6

Created: 2026-10-13T00:48:46Z

Updated: 2026-10-13T00:48:46Z

Source File: data/curated/CrossValidatedPriorLikelihoodConflictDiagnostic.yaml