Reference-family joint error rate post hoc confidence bounds
Mathematical Methods AIMech:ReferenceFamilyJERPostHocConfidenceBounds DRAFT
Aliases: Joint Family-Wise Error Rate Reference-Family Post Hoc Inference, JER-Controlled Reference Family Post Hoc False-Positive Bounds, Blanchard-Neuvial-Roquain Reference-Family Post Hoc Confidence Bounds, Single-Step and Step-Down Lambda-Calibrated Reference-Family JER Control
A user-agnostic post hoc multiple-testing and selective-inference framework introduced by Gilles Blanchard, Pierre Neuvial, and Etienne Roquain (2020, The Annals of Statistics 48(3):1281-1303, arXiv:1703.02307v3/v2, DOI:10.1214/19-AOS1847) that constructs simultaneous level-(1 - alpha) upper confidence bounds V(R) on the number of false positives |R cap H_0(P)| (and complementary lower bounds S(R) = |R| - V(R) on true discoveries |R cap H_1(P)|) valid uniformly over all 2^m candidate subsets R subset N_m = {1, ..., m}. Given a reference family R_ref = (R_k(X), zeta_k(X))_{1 <= k <= K} with controlled Joint Family-Wise Error Rate JER(R_ref, P) = 1 - P_{X~P}(forall k in {1, ..., K}, |R_k(X) cap H_0(P)| <= zeta_k(X)) <= alpha, the framework interpolates from R_ref to arbitrary user-selected sets R via both the worst-case optimal bound V^*_{R_ref}(R) = max_{A in A(R_ref)} |R cap A| and the polynomial-time bound V_bar_{R_ref}(R) = min_{1 <= k <= K} (|R \ R_k| + zeta_k) wedge |R|, proving exact event equivalence E(R_ref, H_0) = {forall R subset N_m, |R cap H_0| <= V_bar_{R_ref}(R)} = {forall R subset N_m, |R cap H_0| <= V^*_{R_ref}(R)} (Proposition 2.1), NP-hardness of V^*_{R_ref}(R) in general (Proposition 2.3), self-consistency idempotence V_bar_{tilde{R_ref}}(R) = V_bar_{R_ref}(R) (Proposition 2.4), and exact optimality V_bar_{R_ref}(R) = V^*_{R_ref}(R) whenever the reference regions R_1 subset ... subset R_K are nested (Proposition 2.5), while calibrating p-value threshold templates t_k(lambda) (linear t_k^L(lambda) = lambda k/m and balanced Beta-quantile t_k^B(lambda) = F_k^{-1}(lambda)) via single-step lambda(alpha, N_m) and step-down Algorithm 1 lambda(alpha, A_hat) under known dependence (LeastFavor, Theorem 5.7) or unknown dependence via finite transformation groups (Rand, Theorem 5.8).
Identity Notes & Concept Scope
No normalized identity match for reference-family joint error rate post hoc confidence bounds across all 442 pinned AIO records and 601 maintained IDs, names, aliases and scoped synonyms. Fixed-string searches across data, sources, research, history, and reviews returned only frontier references in the Round 207 discovery report and review. Distinct from AIMech:SimultaneousPathwiseFalseDiscoveryProportionBounds (Katsevich & Ramdas 2020, The Annals of Statistics 48(6):3465-3487), which constructs simultaneous FDP bounds across a single pre-ordered nested path emptyset = R_0 subset R_1 subset ... subset R_m via Poissonized empirical-process and exponential supermartingale inequalities (and applies closed-testing interpolation only along that path) rather than user-agnostic interpolation over all 2^m arbitrary subsets R subset N_m from a JER-controlled reference family (R_k, zeta_k)_{1 <= k <= K} with single-step and step-down lambda-calibration under LeastFavor and Rand; distinct from AIMech:HadamardDifferentiableThresholdFDPAsymptoticsAndFixedPointDuality (Neuvial 2008), which establishes asymptotic Gaussian CLTs for the FDP of a single data-dependent threshold T(G_hat_m) under independence rather than non-asymptotic simultaneous post hoc bounds P(forall R subset N_m, |R cap H_0(P)| <= V(R)) >= 1 - alpha; and distinct from AIMech:AdaptiveSimesStepUpFDRControlUnderIndependence (Benjamini, Krieger, & Yekutieli 2006) and AIMech:DependencyRobustFalseDiscoveryRateControl (Benjamini & Yekutieli 2001), which control expected FDR = E[FDP] for a single fixed rejection set rather than controlling the joint family-wise error rate JER(R_ref, P) <= alpha to provide post hoc confidence bounds for any post-selection subset R.
Claims & Grounded Evidence
4 Claims
Mechanism
#reference-family-jer-interpolation-equivalence-np-hardness-and-nested-optimality
For any data-dependent reference family R_ref = (R_k(X), zeta_k(X))_{1 <= k <= K} of subsets R_k subset N_m and integers zeta_k, the joint error rate event E(R_ref, H_0) = {forall k = 1, ..., K, |R_k(X) cap H_0| <= zeta_k(X)} is identical as a random event to both {forall R subset N_m, |R cap H_0| <= V^*_{R_ref}(R)} and {forall R subset N_m, |R cap H_0| <= V_bar_{R_ref}(R)} = {S_bar_{R_ref}(H_0) = 0}, where V^*_{R_ref}(R) = max_{A in A(R_ref)} |R cap A| over A(R_ref) = {A subset N_m : forall k, |R_k cap A| <= zeta_k} and V_bar_{R_ref}(R) = min_{1 <= k <= K} (|R \ R_k| + zeta_k) wedge |R| (Section 2.2, Proposition 2.1, Equations 4-9 in v3; Section 2.2, Proposition 2.1, Equations 4-9 and Appendix B.1 in v2). Computing the worst-case bound V^*_{R_ref}(R) for an arbitrary reference family is NP-hard by reduction from Maximum Independent Set on a graph G = (N_m, E) with 2-element edge sets |R_e| = 2, zeta_e = 1, and R = N_m (Section 2.3, Proposition 2.3 in v3; Proposition 2.2 and Appendix B.1 in v2, where on a 3-vertex triangle graph V^*_{R_ref}({1,2,3}) = 1 < 2 = V_bar_{R_ref}({1,2,3})). Replacing zeta_k by the tightened reference bounds tilde{zeta}_k = V_bar_{R_ref}(R_k) = min_j (|R_k \ R_j| + zeta_j) wedge |R_k| <= zeta_k yields the exact self-consistency idempotence V_bar_{tilde{R_ref}}(R) = V_bar_{R_ref}(R) for all R subset N_m (Proposition 2.4 in v3 / Proposition 2.3 in v2), while whenever the reference sets are nested (R_k subset R_{k'} for k <= k'), the polynomial-time bound is exact: V_bar_{R_ref}(R) = V^*_{R_ref}(R) for every R subset N_m (Proposition 2.5 in v3 / Proposition 2.4 and Appendix B.1 in v2), and the only non-empty sets R admissible with respect to V_bar_{R_ref} (satisfying forall R' supsetneq R, V_bar(R') > V_bar(R) and forall R' subsetneq R, S_bar(R') < S_bar(R)) belong to the reference family {R_1, ..., R_K} (Section 8.4 in v3).
Mechanism
#closed-testing-specialization-simes-shortcut-equivalence-and-unbalancedness
The Goeman & Solari (2011) closed-testing post hoc bound V_bar^{GS}_alpha(R) is recovered as a special case of V^*_{R_ref}(R) by taking R_ref = (R_1, ..., R_K) to be the superset-closed collection of intersection hypotheses rejected at local level alpha with zeta_k = |R_k| - 1 (Section 2.2, pp. 6-7 in v3; Section 8.1-8.2, Equations 38-39 in v2), while for Simes-type local tests phi_I(X) = 1{exists i in {1, ..., |I|} : p_{(i:I)} <= c_i^{|I|}} with monotonic critical values c_i^ell <= c_i^k (ell >= k) and c_i^ell <= c_j^ell (i <= j), the polynomial-time Goeman-Solari shortcut V_bar^{GS}_alpha(R) = |R| - (1 + max{S_r, 1 <= r <= |R|}) vee 0 with S_r = max{0 <= s <= r-1 : p_{(r:R)} <= c_{r-s}^m} is algebraically identical to the threshold-based reference-family bound V_bar_{R_ref}(R) = min_{1 <= k <= m} (sum_{i in R} 1{p_i > c_k^m} + k - 1) wedge |R| with R_k = {i in N_m : p_i <= c_k^m} and zeta_k = k - 1 (Section 8.3, Lemmas 8.1-8.2, Equations 40-41 in v2). Under PRDS (with c_m = 1) or arbitrary dependence (with Hommel factor c_m = sum_{i=1}^m 1/i), the canonical Simes reference family R^0_k(X) = {i in N_m : p_i < alpha k / (m c_m)} controls JER at level alpha (Section 4.1, Theorem 4.1, Equations 15-18 in v3), with exact equality JER(R^0, P) = alpha under i.i.d. U(0,1) global null (m_0 = m, c_m = 1), but exhibits severe conservativeness under positive Gaussian equicorrelation (achieving only 0.73 alpha at rho = 0.2 and 0.39 alpha at rho = 0.8 for m = 1000, alpha = 0.2, Table 1) and exponential rank unbalancedness P(p_{(k:m)} <= alpha k/m) <= exp(-(k/4)(1 - alpha - 1/(m+1))^2) where k = 1 alone consumes 1 - (1 - alpha/m)^m = alpha + o(alpha) of the error budget (4.9e-2 at k = 1 vs. 6.6e-6 at k = 5 and 5.8e-93 at k = 100 for m = 1000, alpha = 0.05, Section 4.2-4.3, Tables 1-2 in v3; Appendix A.1, Proposition A.1(iv), Equation 44 in v2).
Mechanism
#threshold-templates-and-single-step-and-step-down-lambda-calibration
For any size-K one-parameter threshold template t_k(lambda), 1 <= k <= K, that is non-decreasing and left-continuous on [0, 1] with t_k(0) = 0 and generalized inverse t_k^{-1}(y) = max{x in [0, 1] : t_k(x) <= y} (Definition 5.1), the JER of R_lambda = ({i in N_m : p_i(X) < t_k(lambda)})_{1 <= k <= K} satisfies JER(R_lambda, P) = P_{X~P}(min_{1 <= k <= K wedge m_0} t_k^{-1}(p_{(k:H_0)}(X)) < lambda) (Lemma 5.2, Equation 20). Any functional lambda(alpha, A) that is set-non-increasing in A (A subset A' implies lambda(alpha, A') <= lambda(alpha, A)) and satisfies P_{X~P}(min_{1 <= k <= K wedge m_0} t_k^{-1}(p_{(k:H_0(P))}(X)) < lambda(alpha, H_0(P))) <= alpha (Definition 5.3, Equations 21-22) guarantees exact level-alpha JER control both for the single-step calibration lambda(alpha, N_m) (Proposition 5.4) and for the step-down fixed point A_hat of Algorithm 1 defined by A^{(0)} = N_m, lambda_j = lambda(alpha, A^{(j-1)}), A^{(j)} = {i in N_m : p_i(X) >= t_1(lambda_j)} until A^{(j)} = A^{(j-1)} (Proposition 5.5; reducing to the Romano-Wolf step-down FWER procedure when K = 1, Remark 5.6). Under known null dependence (LeastFavor: p_i(X) >= q_i(X) for i in H_0(P) with q ~ nu_m independent of P), deterministic quantile calibration lambda(alpha, A) = max{lambda >= 0 : P_{q~nu_m}(min_{1 <= k <= K wedge |A|} t_k^{-1}(q_{(k:A)}) < lambda) <= alpha} is a valid lambda-calibration (Section 5.3, Theorem 5.7, Equations 23-24); under unknown dependence with finite group invariance (Rand: (p_{H_0}(g'.X))_{g' in G} ~ (p_{H_0}(g'.g.X))_{g' in G} for g in G, such as sign-flipping G = {-1, 1}^n), drawing B transformations (g_1 = id, g_2, ..., g_B i.i.d. uniform on G) and setting lambda(alpha, A) = Psi_{(floor(alpha B) + 1)} of Psi(g_j.X, A) = min_{1 <= k <= K wedge |A|} t_k^{-1}(p_{(k:A)}(g_j.X)) controls JER at level floor(alpha B)/B <= alpha for any deterministic template (Section 5.4, Theorem 5.8, Equation 25, and Appendix A, pp. 26-27 in v3).
Mechanism
#linear-vs-balanced-template-asymptotic-conservativeness-and-higher-criticism-detection-optimality
Instantiating the template with the linear family t_k^L(lambda) = lambda k/m yields R^L and R^{L,sd} (Section 6.1, Equations 26-28 in v3), whereas the balanced template t_k^B(lambda) = F_k^{-1}(lambda) with F_k(x) = P_{q~nu_m}(q_{(k:m)} <= x) equalizes marginal k-FWER violation probabilities across k = 1, ..., K under the global null and satisfies alpha/K <= lambda^B(alpha, N_m) <= alpha (Section 6.2, Equations 29-31 in v3; Section 5.2, Equations 29-33 in v2). Under independence with K = m, Proposition A.1 and Lemmas A.2-A.3 (v2 Appendix A.1-A.2, Equations 42-46) prove that t_k^B(lambda) >= k/(m+1) - (k/(m+1)(1 - k/(m+1)))^{1/2} m^{-1/2} (4 log(1/lambda))^{1/2}, that lambda^B(alpha, N_m) <= (log m)^{-1/4} -> 0, and that for fixed pi_0 = m_0/m in (0, 1) the single-step balanced JER collapses at rate P(exists k <= m_0 : q_{(k:m_0)} < t_k^B(lambda^B(alpha))) <= C(pi_0) (log m)^{-1/8} -> 0 because P(p_{(k:m_0)} <= t_k^B(lambda)) <= exp(-(k/32)(1 - m_0/m)^2), making either template truncation K < m or step-down calibration R^{B,sd} essential to recover nominal level alpha (Section 5.3 in v2; Sections 6.2, 7.1, and 8.1 in v3). Furthermore, in the Donoho-Jin (2004) sparse Gaussian mixture H_i ~ B(m^{-beta}), mu_m = sqrt(2 r log m) with beta in (1/2, 1), Theorem 7.2 (v2 Section 7.3, Equations 35-37 and Appendix B.2) proves that the balanced family R^B with K = m achieves optimal detection power Pow^*(R^B, P) = P(S_bar_{R^B}(N_m) >= 1) -> 1 for all r > rho^*(beta) = (beta - 1/2) 1_{(1/2, 3/4]}(beta) + (1 - sqrt(1 - beta))^2 1_{(3/4, 1)}(beta), whereas the linear family R^L (Theorem 7.3 and Appendix B.3) is asymptotically powerless (limsup Pow^*(R^L, P) <= alpha) throughout the sparse-weak regime beta - 1/2 < r < (1 - sqrt(1 - beta))^2.
⚠️ Curated Limitations & Negative Results
- For general non-nested reference families R_ref = (R_k, zeta_k)_{1 <= k <= K} (such as arbitrary closed-testing intersection families with zeta_k = |R_k| - 1), computing the optimal worst-case bound V^*_{R_ref}(R) is NP-hard (Proposition 2.3 in v3 / Proposition 2.2 in v2), and the polynomial-time bound V_bar_{R_ref}(R) can be strictly looser than V^*_{R_ref}(R) (e.g., V_bar = 2 > 1 = V^* on a 3-vertex triangle graph).
- Theorem 5.8 establishes exact finite-sample JER control under unknown dependence (Rand) only when the threshold template t_k(lambda) is deterministic (such as the linear template t_k^L(lambda) = lambda k/m); for the data-dependent balanced template t_k^B(X, lambda) estimated by permutations or sign-flipping, marginal invariance of t_k(g.X, lambda) and p_{H_0}(g.X) does not imply joint invariance of ((t_k(X, lambda))_{1 <= k <= K}, p_{H_0}(X)), leaving an open theoretical gap shared with Theorem 1 (Equation 12) of Meinshausen (2006) (Section 6.2, Remark 6.2, p. 19 in v3; Section 9, p. 25, Footnote 2 in v2).
- Under independence with full template size K = m and non-trivial signal proportion pi_0 < 1, the single-step balanced calibration lambda^B(alpha, N_m) <= (log m)^{-1/4} yields an achieved JER that decays to zero at rate O((log m)^{-1/8}) (Lemmas A.2-A.3 in v2), requiring either step-down calibration R^{B,sd} or smaller template size K < m; conversely, naively updating the balanced template t_k^B(lambda, A^{(j)}) inside the step-down loop can destroy set monotonicity in H_0 (Section 8.2, p. 24 in v3).
- All 29 pages of arXiv:1703.02307v3 and all 34 pages of arXiv:1703.02307v2 were read in full text. Three minor typographical slips in the primary sources were identified and corrected in our verification: (1) Remark 2.2 (v3 p. 8) writes 'for any R in N_m' instead of 'R subset N_m' and 'S_bar(R) := |R| - S_bar(R)' instead of 'S_bar(R) := |R| - V_bar(R)' (as correctly printed in v2 p. 5 Equation 7 and v3 p. 22), (2) Section 7.1 (v3 p. 19) describes the empirical JER violation event as '|R_k(X) cap H_0(P)| > k' instead of '|R_k(X) cap H_0(P)| > zeta_k = k - 1' (i.e. >= k, corresponding to Equations 1, 5, and 13), and (3) Appendix A Proof of Theorem 5.8 (v3 p. 26) writes '1 <= j <= m' instead of '1 <= j <= B' for the index range of Y_j = Psi(g_j.X, H_0) and Psi(g'_j.X, H_0). Constructed numerical checks in research/discovery/reference-family-jer-post-hoc-confidence-bounds-boundaries.py verify: (1) triangle-graph NP-hardness gap V^* = 1 < V_bar = 2, self-consistency idempotence across all 32 subsets, nested exactness V_bar = V^* and Goeman-Solari shortcut equivalence across all 64 subsets, and Section 8.4 admissibility, (2) Table 2 exact Beta tail probabilities (k=1: 0.048772, k=5: 6.5525e-06, k=100: 5.7919e-93) and Proposition A.1 bounds, and (3) Theorems 5.7-5.8 single-step and step-down JER control under known equicorrelation (rho=0.5: JER_Simes=0.085 < JER_ss=0.135 <= JER_sd=0.1817 <= alpha=0.2) and unknown-dependence sign-flipping (B=99: JER_sd=0.1667 <= 0.2).
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:DataDredgingBias
Data Dredging Bias
|
related_match | Reference-family JER control provides user-agnostic post hoc upper bounds V(R) on false positives valid simultaneously over all 2^m subsets R subset N_m, directly preventing selection-effect and data-snooping invalidation (AIO:DataDredgingBias). |
Evidence Sources & Primary Literature
4 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
bnr2020-v3-pdf
|
Gilles Blanchard, Pierre Neuvial, Etienne Roquain
|
primary paper
full_text
|
ID: https://doi.org/10.1214/19-AOS1847
sha256:7711fc4b689e…
|
bnr2020-v2-pdf
|
Gilles Blanchard, Pierre Neuvial, Etienne Roquain
|
primary paper
full_text
|
ID: https://doi.org/10.48550/arXiv.1703.02307
sha256:a14025460c20…
|
bnr2020-crossref
|
Gilles Blanchard, Pierre Neuvial, Etienne Roquain
|
primary paper
abstract_only
|
ID: https://doi.org/10.1214/19-AOS1847
sha256:2420271bd9a5…
|
bnr2020-abs
|
Gilles Blanchard, Pierre Neuvial, Etienne Roquain
|
primary paper
abstract_only
|
ID: https://doi.org/10.48550/arXiv.1703.02307
sha256:91c73d6251c4…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini 3
Created: 2026-10-17T16:55:00Z
Updated: 2026-10-17T16:55:00Z
Source File:
data/curated/ReferenceFamilyJERPostHocConfidenceBounds.yaml
· Schema:
curated_concept.yaml
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories