OLS Post-Lasso and Fitness-Thresholded Post-Model-Selection Estimation in High-Dimensional Sparse Models

A two-stage high-dimensional sparse regression methodology and non-asymptotic oracle rate theory (introduced by Alexandre Belloni and Victor Chernozhukov in 2013) that refits Ordinary Least Squares (OLS) on the covariate support $\widehat{T} \subseteq \{1, \dots, p\}$ selected by a first-stage $\ell_1$-penalized Lasso estimator $\widehat{\beta}_{\mathrm{L}} \in \arg\min_{\beta \in \mathbb{R}^p} \left\{\widehat{Q}(\beta) + \frac{\lambda}{n}\|\beta\|_1\right\}$ ($\widehat{Q}(\beta) = \frac{1}{n}\sum_{i=1}^n (y_i - x_i'\beta)^2$), or on an augmented/fitness-thresholded support $\widetilde{T} = \widehat{T}_{t_\gamma} = \{j \in \operatorname{supp}(\widehat{\beta}_{\mathrm{L}}) : |\widehat{\beta}_{\mathrm{L},j}| > t_\gamma\}$ with $\widetilde{\beta}_{\mathrm{PL}} \in \arg\min_{\operatorname{supp}(\beta) \subseteq \widetilde{T}} \widehat{Q}(\beta)$ where $t_\gamma = \min\{t \ge 0 : \widehat{Q}(\widehat{\beta}_{\mathrm{PL}}(\widehat{T}_t)) - \widehat{Q}(\widehat{\beta}_{\mathrm{L}}) \le \gamma\}$ (Equation 2.11). In the general nonparametric approximation model $y_i = f(z_i) + \epsilon_i = x_i'\beta_0 + r_i + \epsilon_i$ with approximation error $c_s = \left(\mathbb{E}_n[r_i^2]\right)^{1/2}$, Theorem 3 bounds the random empirical sparsity $\widehat{m} = |\widetilde{T} \setminus T| \le s\,L_n$ via the sublinearity of maximal sparse eigenvalues $\phi(\ell k) \le \lceil\ell\rceil\phi(k)$ (Lemma 3), and Theorem 4 proves that OLS Post-Lasso achieves the prediction error bound $\|\widetilde{f}_{\mathrm{PL}} - f\|_{2,n} \lesssim \sigma\sqrt{\frac{\widehat{m}\log p + k\log(ep/k)}{n\,\kappa(k + \widehat{m})}} + c_s + \mathbf{1}\{\widehat{T} \not\supseteq T\}\sqrt{\left[\widehat{Q}(\widetilde{\beta}_{\mathrm{PL}}) - \widehat{Q}(\widehat{\beta}_{\mathrm{L}})\right]_+} + B_n\,\sigma\sqrt{\frac{s\log p}{n}}\frac{1}{\kappa(s, \mathbf{c})}$, matching Lasso's rate even when model selection partially fails and achieving a strictly faster $\sqrt{(s/n)\log(ep/s)}$ rate when $\widehat{T} = T$, while completely eliminating the unavoidable $\ell_1$-shrinkage bias $\|\widehat{f}_{\mathrm{L}} - f\|_{2,n} \ge \frac{\|\widehat{\beta}_{\mathrm{L}}\|_1}{\|f\|_{2,n}} \cdot \frac{(1 - 1/c)\lambda}{n} - c_s$ proved in Remark 3.1 (Equation 3.19).