Factor-Adjusted Regularized Model Selection and Decorrelated Variable Screening
Algorithms AIMech:FactorAdjustedRegularizedModelSelectionAndDecorrelatedVariableScreening DRAFT
Aliases: FarmSelect, Factor-Adjusted Regularized Model Selection, Decorrelated Variable Screening, Fan-Ke-Wang Factor-Adjusted Lasso and GLM Selection
A two-step high-dimensional model-selection and variable-screening methodology (introduced by Jianqing Fan, Yuan Ke, and Kaizheng Wang in 2020 as *FarmSelect*) for sparse generalized linear models and convex M-estimation under strong cross-sectional covariate dependence driven by latent common factors $\mathbf{x}_t = \mathbf{B}\mathbf{f}_t + \mathbf{u}_t$ ($t = 1, \dots, n$, $p \gg n$). Because pervasive common factors spike the top $K$ eigenvalues of $\operatorname{cov}(\mathbf{x}_t)$ at rate $O(p)$ and cause raw covariates $\mathbf{X}$ to violate the Lasso irrepresentable condition $\|\mathbf{X}_{S^c}^T\mathbf{X}_S(\mathbf{X}_S^T\mathbf{X}_S)^{-1}\|_\infty \le 1 - \tau$, Step 1 estimates the factor number $\widehat{K} = \arg\max_{1 \le k \le k_{\max}} \lambda_k(\mathbf{X}\mathbf{X}^T) / \lambda_{k+1}(\mathbf{X}\mathbf{X}^T)$ (Equation 3.6) and extracts the principal-component factor matrix $\widehat{\mathbf{F}} \in \mathbb{R}^{n \times \widehat{K}}$ ($\widehat{\mathbf{F}}^T\widehat{\mathbf{F}} / n = \mathbf{I}_{\widehat{K}}$), loadings $\widehat{\mathbf{B}} = n^{-1}\mathbf{X}^T\widehat{\mathbf{F}}$, and decorrelated idiosyncratic residuals $\widehat{\mathbf{U}} = (\mathbf{I}_n - \widehat{\mathbf{P}})\mathbf{X}$ where $\widehat{\mathbf{P}} = n^{-1}\widehat{\mathbf{F}}\widehat{\mathbf{F}}^T$. Exploiting the identity $\mathbf{x}_t^T\boldsymbol{\beta}^* = \mathbf{u}_t^T\boldsymbol{\beta}^* + \mathbf{f}_t^T\boldsymbol{\gamma}^*$ with $\boldsymbol{\gamma}^* = \mathbf{B}^T\boldsymbol{\beta}^*$ (Lemma 3.1, p. 8), Step 2 solves the factor-augmented profile penalized problem $\widehat{\boldsymbol{\beta}} \in \arg\min_{\boldsymbol{\beta} \in \mathbb{R}^p, \boldsymbol{\gamma} \in \mathbb{R}^{\widehat{K}}} L_n(\mathbf{y}, \widehat{\mathbf{U}}\boldsymbol{\beta} + \widehat{\mathbf{F}}\boldsymbol{\gamma}) + p_\lambda(\boldsymbol{\beta})$ penalizing only $\boldsymbol{\beta}$ (Equation 3.2), which for linear models reduces exactly to projected Lasso $\arg\min_{\boldsymbol{\beta}} \frac{1}{2n}\|(\mathbf{I}_n - \widehat{\mathbf{P}})(\mathbf{y} - \widehat{\mathbf{U}}\boldsymbol{\beta})\|_2^2 + p_\lambda(\boldsymbol{\beta})$ (Equation 3.5), while *Decorrelated Variable Screening* ranks features by their marginal coefficient $\widetilde{\beta}_j$ in the factor-augmented regression of $\mathbf{y}$ on $(\widehat{\mathbf{U}}_j, \widehat{\mathbf{F}})$ (Equation 3.7).
Identity Notes & Concept Scope
Resolved against all 442 pinned AIO concepts and 670 maintained AIMech concepts. Distinct from `AIMech:PrincipalOrthogonalComplementThresholdingForApproximateFactorCovarianceEstimation` (Fan, Liao, & Mincheva 2013, which thresholds the residual covariance matrix $\widehat{\mathbf{R}}_K$ for unsupervised covariance and precision matrix estimation rather than decomposing the linear predictor $\mathbf{x}_t^T\boldsymbol{\beta}^* = \mathbf{u}_t^T\boldsymbol{\beta}^* + \mathbf{f}_t^T\boldsymbol{\gamma}^*$ to restore the irrepresentable condition and model-selection sign consistency in supervised generalized linear models), distinct from `AIMech:PreconditionedLassoAndPaulilassoViaSingularValueDecomposition` (Jia & Rohe 2015, which left-multiplies $(\mathbf{y}, \mathbf{X})$ by $\mathbf{U}\mathbf{D}^{-1}\mathbf{U}^T$ when $p > n$ rather than projecting out the top $K$ latent common factors into an unpenalized nuisance space $\widehat{\mathbf{F}}\boldsymbol{\gamma}$), and distinct from `AIMech:SureIndependenceScreeningAndIteratedSureIndependenceScreeningForUltrahighDimensionalFeatureSpace` (Fan & Lv 2008, which screens raw marginal correlations $\mathbf{X}^T\mathbf{y}$ without removing factor-induced confounding).
Claims & Grounded Evidence
3 Claims
Mechanism
#factor-augmented-predictor-decomposition-and-projected-lasso-equivalence
Under the approximate factor model $\mathbf{x}_t = \mathbf{B}\mathbf{f}_t + \mathbf{u}_t$ and sparse generalized linear model density $f(y_t \mid \mathbf{x}_t, \boldsymbol{\beta}^*) = \exp\{\frac{y_t(\beta_0^* + \mathbf{x}_t^T\boldsymbol{\beta}^*) - b(\beta_0^* + \mathbf{x}_t^T\boldsymbol{\beta}^*)}{a(\phi)} + c(y_t, \phi)\}$ (Section 2.2, Equation 2.3), Lemma 3.1 (Section 3.1, p. 8) proves that $\mathbf{u}_t$ satisfies the augmented GLM $f(y_t \mid \mathbf{u}_t, \mathbf{f}_t, \boldsymbol{\beta}^*, \boldsymbol{\gamma}^*) = \exp\{\frac{y_t(\mathbf{u}_t^T\boldsymbol{\beta}^* + \mathbf{f}_t^T\boldsymbol{\gamma}^*) - b(\mathbf{u}_t^T\boldsymbol{\beta}^* + \mathbf{f}_t^T\boldsymbol{\gamma}^*)}{a(\phi)} + c(y_t, \phi)\}$ with $\boldsymbol{\gamma}^* = \mathbf{B}^T\boldsymbol{\beta}^*$. In the linear case $b(\theta) = \theta^2/2$ (Section 3.2, pp. 9–10, Equations 3.4–3.5), profiling out the unpenalized factor coefficient $\boldsymbol{\gamma} \in \mathbb{R}^{\widehat{K}}$ in $\min_{\boldsymbol{\beta}, \boldsymbol{\gamma}} \frac{1}{2n}\|\mathbf{y} - \widehat{\mathbf{U}}\boldsymbol{\beta} - \widehat{\mathbf{F}}\boldsymbol{\gamma}\|_2^2 + p_\lambda(\boldsymbol{\beta})$ yields the exact orthogonal-projection identity $\widehat{\boldsymbol{\beta}} \in \arg\min_{\boldsymbol{\beta} \in \mathbb{R}^p} \frac{1}{2n}\|(\mathbf{I}_n - \widehat{\mathbf{P}})(\mathbf{y} - \widehat{\mathbf{U}}\boldsymbol{\beta})\|_2^2 + p_\lambda(\boldsymbol{\beta})$ where $\widehat{\mathbf{P}} = \widehat{\mathbf{F}}(\widehat{\mathbf{F}}^T\widehat{\mathbf{F}})^{-1}\widehat{\mathbf{F}}^T$, and since $\widehat{\mathbf{F}}^T\widehat{\mathbf{U}} = \mathbf{0}$, $(\mathbf{I}_n - \widehat{\mathbf{P}})\widehat{\mathbf{U}} = \widehat{\mathbf{U}}$.
Scope
#asymptotic-restoration-of-irrepresentable-condition-and-glm-sign-consistency
Let $\mathbf{W} = (\mathbf{U}, \mathbf{F}) \in \mathbb{R}^{n \times (p+K)}$ and $\widehat{\mathbf{W}} = (\widehat{\mathbf{U}}, \widehat{\mathbf{F}})$ with augmented active set $\bar{S} = S \cup \{p+1, \dots, p+K\}$ of cardinality $\bar{s} = |S| + K$. Under Assumptions 4.1–4.2 (requiring the Hessian $\nabla^2 L_n(\mathbf{y}, \mathbf{W}_{\bar{S}}\boldsymbol{\theta}_{\bar{S}}^*)$ to have minimum eigenvalue $\ge \kappa_0$ and the population idiosyncratic design $\mathbf{W}$ to satisfy the irrepresentable condition $\|\nabla_{\bar{S}^c,\bar{S}}^2 L_n(\mathbf{y}, \mathbf{W}\boldsymbol{\theta}^*)(\nabla_{\bar{S},\bar{S}}^2 L_n(\mathbf{y}, \mathbf{W}\boldsymbol{\theta}^*))^{-1}\|_\infty < 1 - 2\tau$) and factor estimation bound $\max_{k,t}|\widehat{F}_{tk} - F_{tk}| \lesssim \sqrt{\log n / p} + 1/\sqrt{n}$ and $\max_{j,t}|\widehat{U}_{tj} - U_{tj}| \lesssim (\sqrt{\log p / n} + 1/\sqrt{p})\sqrt{\log n}$ (Assumption 4.3, Theorem 4.2, p. 15), if $\omega_n = \bar{s}(\sqrt{\frac{\log p}{n}} + \frac{1}{\sqrt{p}})\sqrt{\log n} \lesssim \kappa_0 / \tau$, then Theorem 4.1 (Section 4.1, p. 14) proves that FarmSelect achieves exact sign consistency $\operatorname{sign}(\widehat{\boldsymbol{\beta}}) = \operatorname{sign}(\boldsymbol{\beta}^*)$ and error rates $\|\widehat{\boldsymbol{\beta}}_S - \boldsymbol{\beta}_S^*\|_\infty = O_p(\sqrt{\bar{s}}(\sqrt{\log p / n} + 1/\sqrt{p}))$ and $\|\widehat{\boldsymbol{\beta}}_S - \boldsymbol{\beta}_S^*\|_2 = O_p(\bar{s}(\sqrt{\log p / n} + 1/\sqrt{p}))$.
Mechanism
#decorrelated-marginal-variable-screening-and-eigenvalue-ratio-factor-selection
Section 3.3 (p. 11, Equation 3.6) determines the latent factor dimension $\widehat{K} = \arg\max_{1 \le k \le k_{\max}} \lambda_k(\mathbf{X}\mathbf{X}^T) / \lambda_{k+1}(\mathbf{X}\mathbf{X}^T)$ via the Ahn–Horenstein eigenvalue ratio estimator. For ultrahigh-dimensional regimes where $p$ is too large for direct penalized optimization, Section 3.4 (pp. 11–12, Equation 3.7) defines Decorrelated Variable Screening by fitting $p$ marginal factor-augmented regressions $(\widetilde{\beta}_j, \widetilde{\boldsymbol{\gamma}}_j) = \arg\min_{\beta_j \in \mathbb{R}, \boldsymbol{\gamma}_j \in \mathbb{R}^{\widehat{K}}} L_n(\mathbf{y}, \widehat{\mathbf{U}}_j\beta_j + \widehat{\mathbf{F}}\boldsymbol{\gamma}_j)$ for $1 \le j \le p$ and retaining variables with the largest $|\widetilde{\beta}_j|$, eliminating the spurious marginal correlations induced by common factors $\mathbf{F}$ that cause standard Sure Independence Screening on raw $\mathbf{X}_j$ to select inactive factor-loaded covariates.
⚠️ Curated Limitations & Negative Results
- Sign consistency requires that the idiosyncratic components u_t = x_t - B f_t satisfy the population irrepresentable condition ||nabla^2_{S^c,S} L_n (nabla^2_{S,S} L_n)^{-1}||_infty < 1 - 2*tau (Assumption 4.2) and that the scaling omega_n = s_bar * (sqrt(log p / n) + 1/sqrt(p)) * sqrt(log n) is sufficiently small (Theorem 4.1); if strong correlations persist within u_t after removing the top K factors, L1 selection bias can remain.
- Step 1 PCA factor recovery requires pervasiveness of the K factors across p -> infty covariates (||p^{-1} B^T B - Sigma_B||_2 -> 0 with lambda_min(Sigma_B) > 0, Assumption 2.1); weak or local factors affecting only o(p) covariates are not separated from u_t by top-K eigen-decomposition.
- Primary-source typographical audit of Fan, Ke, & Wang (2020, arXiv:1612.08490v2) uncovered four minor notation and typographical issues that are corrected in this record: (1) Section 1 p. 2 line 3 spells 'redundent coefficients', p. 3 line 11 writes 'overcome the the aforementioned problems' (duplicated 'the'), and p. 4 line 15 spells 'facilitate the implemention our method'; (2) Section 3.1 p. 8 line 9 writes 'The theoretical for us to ignore the constraint' (missing 'justification'), and Lemma 3.1 p. 8 has an unmatched opening parenthesis in '(eta_t = y_t - b''((1, x_t^T)beta*)'; (3) Section 3.2 p. 9 line 13 writes 'Define B_hat_0 = (0_K, B_hat^T)^T in R^{p x K} and ... X_1 = F_hat B_hat_0 + U_hat_1', which is dimensionally incompatible (n x K times p x K) and inconsistent with Section 3.1 p. 7 where B_0 = (0_K, B^T)^T in R^{K x p} and X_1 = F B_0^T + U_1; and (4) Section 5.1 p. 17 step (1) writes 'f_tilde_t = (lambda_1^{-1/2} xi_1^T z_t, lambda_2^{-1/2} xi_2^T z_t, lambda_3^{-1/2} xi_1^T z_t)^T' with 'xi_1^T' instead of 'xi_3^T' in the third coordinate.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:LassoRegression
Lasso Regression
|
related_match | FarmSelect restores the irrepresentable condition and sign consistency of L1-penalized Lasso and GLM estimators by profiling out principal-component common factors. |
AIO:PrincipalComponentAnalysis
Principal Component Analysis
|
related_match | Step 1 extracts latent factors F_hat and idiosyncratic residuals U_hat via principal component analysis of the covariate design matrix X. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
fkw2020-farmselect-pdf
|
Jianqing Fan, Yuan Ke, Kaizheng Wang
|
primary paper
full_text
|
ID: https://doi.org/10.1016/j.jeconom.2020.01.006
sha256:76af74ebf0f1…
|
fkw2020-farmselect-abs
|
Jianqing Fan, Yuan Ke, Kaizheng Wang
|
primary paper
abstract_only
|
ID: https://arxiv.org/abs/1612.08490
sha256:3f90ff509f73…
|
fkw2020-farmselect-crossref
|
Jianqing Fan, Yuan Ke, Kaizheng Wang
|
primary paper
abstract_only
|
ID: https://doi.org/10.1016/j.jeconom.2020.01.006
sha256:9daa4513400e…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T05:04:00Z
Updated: 2026-10-19T05:04:00Z
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories