AIMech AI

Heteroskedastic PCA Principal Subspace Confidence Regions and Spiked Covariance Inference with Missing Data

Mathematical Methods AIMech:HeteroskedasticPCAPrincipalSubspaceConfidenceRegionsAndSpikedCovarianceInferenceWithMissingData DRAFT
Aliases: Diagonal-Deleted Heteroskedastic PCA Subspace Inference with Missing Data, Yan-Chen-Fan Heteroskedastic PCA Confidence Regions and Spiked Covariance Inference, Deflated Gram Matrix Principal Subspace Wald Inference
A distributional inference and uncertainty quantification methodology (introduced by Yuling Yan, Yuxin Chen, and Jianqing Fan in 2024) for estimating and constructing valid confidence regions for the $r$-dimensional principal subspace $\boldsymbol{U}^\star \in \mathbb{O}^{d \times r}$ and linear forms of eigenvectors of a low-rank spiked covariance matrix $\boldsymbol{\Sigma}^\star = \boldsymbol{U}^\star\boldsymbol{\Lambda}^\star\boldsymbol{U}^{\star\top} \in \mathbb{R}^{d \times d}$ from $n$ incomplete sample columns observed at rate $p$ with entrywise heteroskedastic noise variances $\sigma_{i,t}^2 = \mathrm{Var}(E_{i,t})$. Because the diagonal of the inverse-probability-weighted empirical Gram matrix $\widehat{\boldsymbol{G}} = \frac{1}{n p^2}\mathcal{P}_\Omega(\boldsymbol{X})\mathcal{P}_\Omega(\boldsymbol{X})^\top$ has severe bias $\mathbb{E}[\widehat{G}_{i,i}] = \frac{1}{p}\Sigma_{i,i}^\star + \frac{1}{n p}\sum_{t=1}^n \sigma_{i,t}^2$, the method eliminates diagonal bias either by zeroing out the diagonal $\boldsymbol{M} = \mathcal{P}_{\mathrm{off-diag}}(\widehat{\boldsymbol{G}})$ (Algorithm 1, Deflated Heteroskedastic PCA) or by iterative low-rank diagonal imputation (HeteroPCA). Theorems 1–4 establish a second-order Neumann-series expansion showing that for every row $1 \le i \le d$, the aligned subspace row error $(\boldsymbol{U}\boldsymbol{H} - \boldsymbol{U}^\star)_{i,\cdot}^\top$ is asymptotically normal $\mathcal{N}(\boldsymbol{0}, \boldsymbol{S}^{(i,\star)})$, that the plug-in covariance estimator $\widehat{\boldsymbol{S}}^{(i)}$ (Equations 3.8–3.10) yields pivotal Wald statistics $W_i = (\boldsymbol{U}\widehat{\boldsymbol{H}} - \boldsymbol{U}^\star)_{i,\cdot}(\widehat{\boldsymbol{S}}^{(i)})^{-1}(\boldsymbol{U}\widehat{\boldsymbol{H}} - \boldsymbol{U}^\star)_{i,\cdot}^\top \stackrel{\mathrm{d}}{\to} \chi_r^2$, and that linear forms $\boldsymbol{c}^\top(\boldsymbol{u}_k - \boldsymbol{u}_k^\star)$ (Theorems 5–6) and entrywise covariance estimates $\widehat{\Sigma}_{i,j} - \Sigma_{i,j}^\star$ (Theorems 7–8) achieve valid $(1-\alpha)$ Gaussian confidence intervals without requiring eigengaps among the top $r$ eigenvalues for subspace and covariance inference.
Identity Notes & Concept Scope Resolved against all 442 pinned AIO concepts and 687 maintained AIMech concepts. Distinct from `AIMech:AsymmetricLowRankEigenvalueAndLinearFormEigenvectorPerturbationAnalysis` (Chen, Cheng, & Fan 2021, which analyzes a single $n \times n$ asymmetric noisy matrix with independent entries rather than sample Gram matrices $\frac{1}{n}\sum_{t=1}^n \boldsymbol{z}_t\boldsymbol{z}_t^\top$ whose entries have dependent Wishart-type products across $n$ observations, diagonal noise-variance inflation, and multi-rank $\chi_r^2$ Wald subspace confidence regions) and from `AIMech:DebiasedNoisyMatrixCompletionInferenceAndUncertaintyQuantification` (Chen, Fan, Ma, & Yan 2019, which addresses homoskedastic matrix completion where $d_1 \asymp d_2$ rather than spiked covariance PCA with $d \ll n$ or $d \gtrsim n$ under heteroskedastic column observations).

Claims & Grounded Evidence

3 Claims
Mechanism #claim-diagonal-deletion-bias-elimination-and-rowwise-subspace-normality
Zeroing the diagonal of the inverse-probability-weighted sample Gram matrix $\boldsymbol{M} = \mathcal{P}_{\mathrm{off-diag}}(\frac{1}{n p^2}\mathcal{P}_\Omega(\boldsymbol{X})\mathcal{P}_\Omega(\boldsymbol{X})^\top)$ removes the $\frac{1}{np}\sum_{t=1}^n\sigma_{i,t}^2$ diagonal noise spike while incurring only an $O(\mu r/d)$ deterministic off-diagonal perturbation, and Theorem 1 proves that under $\sqrt{n}p \gg \kappa^2\mu r\sqrt{\log^3(d+n)}$ the aligned principal subspace rows satisfy $(\boldsymbol{U}\widehat{\boldsymbol{H}} - \boldsymbol{U}^\star)_{i,\cdot}^\top = (\boldsymbol{\Lambda}^\star)^{-1}\boldsymbol{U}^{\star\top}(\boldsymbol{M} - \mathbb{E}[\boldsymbol{M}])_{\cdot,i} + \boldsymbol{\psi}_i$ with $(\boldsymbol{S}^{(i,\star)})^{-1/2}(\boldsymbol{U}\widehat{\boldsymbol{H}} - \boldsymbol{U}^\star)_{i,\cdot}^\top \stackrel{\mathrm{d}}{\to} \mathcal{N}(\boldsymbol{0},\boldsymbol{I}_r)$ (Sections 2.2 and 3.1, Equations 2.5 and 3.2–3.7, Theorem 1).
Mechanism #claim-plug-in-covariance-consistency-and-chi-square-subspace-wald-regions
Using the data-driven variance-covariance estimator $\widehat{\boldsymbol{S}}^{(i)} = \frac{1}{n}\boldsymbol{\Lambda}^{-1}\boldsymbol{U}^\top\widehat{\boldsymbol{\Xi}}^{(i)}\boldsymbol{U}\boldsymbol{\Lambda}^{-1}$ with $\widehat{\Xi}_{j,l}^{(i)} = \mathbb{I}_{\{j=l\ne i\}}\frac{1}{n}\sum_{t=1}^n Z_{i,t}^2 Z_{j,t}^2$ (Equations 3.8–3.10), Theorem 2 proves $\|(\widehat{\boldsymbol{S}}^{(i)})^{-1/2}\boldsymbol{S}^{(i,\star)}(\widehat{\boldsymbol{S}}^{(i)})^{-1/2} - \boldsymbol{I}_r\| = o_P(1)$ and establishes the pivotal $\chi_r^2$ convergence $W_i = (\boldsymbol{U}\widehat{\boldsymbol{H}} - \boldsymbol{U}^\star)_{i,\cdot}(\widehat{\boldsymbol{S}}^{(i)})^{-1}(\boldsymbol{U}\widehat{\boldsymbol{H}} - \boldsymbol{U}^\star)_{i,\cdot}^\top \stackrel{\mathrm{d}}{\to} \chi_r^2$ without requiring any separation among the $r$ nonzero eigenvalues $\lambda_1^\star,\dots,\lambda_r^\star$ (Section 3.1, Theorem 2, Corollary 1, Section 3.2, Theorems 3–4).
Application #claim-individual-eigenvector-linear-forms-and-entrywise-covariance-inference
When the $k$-th eigenvalue $\lambda_k^\star$ is separated by $\Delta_k^\star = \min_{l \ne k}|\lambda_k^\star - \lambda_l^\star| > 0$, Theorems 5–6 establish a second-order resolvent expansion for any linear form $\boldsymbol{c}^\top(\boldsymbol{u}_k - \boldsymbol{u}_k^\star) / \sqrt{\widehat{v}_{\boldsymbol{c},k}} \stackrel{\mathrm{d}}{\to} \mathcal{N}(0,1)$ Incorporating cross-subspace interaction terms $\sum_{l \ne k}\frac{\lambda_l^\star}{\lambda_k^\star - \lambda_l^\star}\boldsymbol{u}_l^\star\boldsymbol{u}_l^{\star\top}$, while Theorems 7–8 prove eigengap-free asymptotic normality $(\widehat{\Sigma}_{i,j} - \Sigma_{i,j}^\star)/\sqrt{\widehat{\vartheta}_{i,j}} \stackrel{\mathrm{d}}{\to} \mathcal{N}(0,1)$ for the reconstructed covariance matrix $\widehat{\boldsymbol{\Sigma}} = \boldsymbol{U}\boldsymbol{\Lambda}\boldsymbol{U}^\top$ (Sections 3.3–3.4, Theorems 5–8).

⚠️ Curated Limitations & Negative Results

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes
AIO Target Class Mapping Relation Mapping Rationale
AIO:PrincipalComponentAnalysis
Principal Component Analysis
narrow_match Provides bias-corrected principal subspace estimation and asymptotic chi-square/Gaussian confidence regions for PCA under heteroskedastic noise and missing data.
AIO:DimensionalityReduction
Dimensionality Reduction
related_match Quantifies uncertainty of low-dimensional principal subspace embeddings from incomplete heteroskedastic observations.

Evidence Sources & Primary Literature

3 Sources
Source ID Publication / Document Type / Access Reference & Checksum
ycf2024-hpca-pdf
Yuling Yan, Yuxin Chen, Jianqing Fan
primary paper
full_text
ID: https://doi.org/10.1214/24-AOS2366
sha256:2aaf20e12711…
ycf2024-hpca-abs
Yuling Yan, Yuxin Chen, Jianqing Fan
primary paper
abstract_only
ID: https://arxiv.org/abs/2107.12365
sha256:d92dcee36dd4…
ycf2024-hpca-cr
Yuling Yan, Yuxin Chen, Jianqing Fan
primary paper
abstract_only
ID: https://doi.org/10.1214/24-AOS2366
sha256:9d30bd3ca31b…

Ontology & Grounding Topology

Open in interactive network graph →
CONCEPT KIND Mathematical Methods AIMech:HeteroskedasticPCAPrincipalSubspaceConfidenceRegionsAndSpikedCovarianceInferenceWithMissingData Heteroskedastic PCA... 3 claims · 3 sources AIO GROUNDING AIO:PrincipalComponentAnalysis (2)

Curation Provenance

Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T05:38:00Z
Updated: 2026-10-19T05:38:00Z
Source File: data/curated/HeteroskedasticPCAPrincipalSubspaceConfidenceRegionsAndSpikedCovarianceInferenceWithMissingData.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories