Scaled LASSO and Huber Concomitant Penalized Joint Sparse Regression
Algorithms AIMech:ScaledLassoAndHuberConcomitantPenalizedJointSparseRegression DRAFT
Aliases: Scaled Lasso, Scaled Sparse Linear Regression, Huber Concomitant Scale-Regression Joint L1 Penalized Estimator, Alternating Residual-Scale Equilibrium LASSO
Scaled LASSO and Huber Concomitant Penalized Joint Sparse Regression is a high-dimensional sparse linear regression and noise-level estimation framework introduced by Sun and Zhang (2012, Biometrika 99(4):879-898, DOI:10.1093/biomet/ass043, arXiv:1104.4595v2) that simultaneously estimates the regression vector $\beta \in \mathbb{R}^p$ and noise standard deviation $\sigma > 0$ via the alternating equilibrium mapping $\widehat{\sigma} = \{|y - X\widehat{\beta}|_2^2 / n(1-a)\}^{1/2}$, $\lambda = \widehat{\sigma}\lambda_0$, and $\widehat{\beta} = \arg\min_\beta \{\frac{1}{2n}|y - X\beta|_2^2 + \lambda\rho(\beta)\}$ ($a \ge 0$, Eqs. 3-4). For $\rho(\beta) = |\beta|_1$, this equilibrium is the global minimizer of the jointly convex Huber (1981) concomitant penalized loss $L_{\lambda_0}(\beta, \sigma) = \frac{|y - X\beta|_2^2}{2n\sigma} + \frac{(1-a)\sigma}{2} + \lambda_0|\beta|_1$ (Proposition 1, Eqs. 5-8), whose profile objective satisfies $\frac{\partial}{\partial\sigma} L_{\lambda_0}(\widehat{\beta}(\sigma), \sigma) = \frac{1-a}{2} - \frac{|y - X\widehat{\beta}(\sigma)|_2^2}{2n\sigma^2}$. With the scale-free universal penalty $\lambda_0 = A\{(2/n)\log(p/\epsilon)\}^{1/2}$ ($A > \{(1-a)/(1-\tau_*)\}^{1/2}$), Scaled LASSO achieves near-oracle prediction, $\ell_q$ parameter estimation (Theorem 1, Corollary 1), quadratic relative noise-level error $\max(1 - \widehat{\sigma}/\sigma^*, 1 - \sigma^*/\widehat{\sigma}) \le \tau_*^2 \lesssim |\beta^*|_0(\log p)/n$, and asymptotic normality $n^{1/2}(\widehat{\sigma}/\sigma - 1) \xrightarrow{d} N(0, 1/2)$ when $|\beta^*|_0(\log p)/n^{1/2} \to 0$ (Theorem 2, Corollary 2), while extending to concave penalties (MCP/SCAD) and Least-Squares after Model Selection (LSE, Theorem 3).
Identity Notes & Concept Scope
Distinct from Square-Root LASSO (`AIMech:SquareRootLassoPivotalConicSparseRegression`), which minimizes the non-separable root-MSE criterion $\{\widehat{Q}(\beta)\}^{1/2} + (\lambda/n)\|\beta\|_1$ via second-order cone programming without an explicit concomitant scale variable $\sigma$ or degrees-of-freedom adjustment $1-a$ (note that profiling $\sigma = |y - X\beta|_2 / \{n(1-a)\}^{1/2}$ out of $L_{\lambda_0}(\beta, \sigma)$ yields $(1-a)^{1/2}\{|y - X\beta|_2 / n^{1/2}\} + \lambda_0|\beta|_1$). Also distinct from standard LASSO (`AIO:LassoRegression`), whose penalty $\lambda$ depends on unknown $\sigma$, and from Städler-Bühlmann-van de Geer (2010) penalized Gaussian negative log-likelihood $\min_{\beta,\sigma} \{\log\sigma + |y - X\beta|_2^2/(2n\sigma^2) + \lambda_0|\beta|_1/\sigma\}$, whose KKT identity $n^{-1}|y - X\widehat{\beta}_{\text{pmle}}|_2^2 + \lambda_0\widehat{\sigma}_{\text{pmle}}|\widehat{\beta}_{\text{pmle}}|_1 = \widehat{\sigma}_{\text{pmle}}^2$ (Eq. 9) introduces a linear first-order shrinkage bias in $\widehat{\sigma}_{\text{pmle}}$ when $\lambda_0|\beta|_1 = \Omega(n^{-1/2})$ (Section 2.2, p. 4), whereas Scaled LASSO's KKT residual-square sandwich $|y - X\beta^*|_2^2 - n\lambda_0\widehat{\sigma}|\beta^*|_1 \le n(1-a)\widehat{\sigma}^2 \le |y - X\beta^*|_2^2 + n\lambda_0\widehat{\sigma}|\beta^*|_1$ (Eq. A6) cancels the first-order bias via the inner product $2\langle y - X\beta^*, X(\widehat{\beta} - \beta^*)\rangle$ and achieves quadratic concentration $\tau_*^2 = O_P(|\beta^*|_0(\log p)/n)$.
Claims & Grounded Evidence
3 Claims
Mechanism
#huber-concomitant-joint-convexity-and-alternating-profile-equilibrium
For the $\ell_1$ penalty $\rho(\beta) = |\beta|_1$ and any $a \in [0, 1)$, the concomitant objective $L_{\lambda_0}(\beta, \sigma) = \frac{|y - X\beta|_2^2}{2n\sigma} + \frac{(1-a)\sigma}{2} + \lambda_0|\beta|_1$ is jointly convex in $(\beta, \sigma)$ on $\mathbb{R}^p \times (0, \infty)$, the profile objective $L_{\lambda_0}(\widehat{\beta}(\sigma), \sigma)$ along the standard LASSO solution path $\widehat{\beta}(\sigma) = \arg\min_\beta L_{\lambda_0}(\beta, \sigma)$ is strictly convex in $\sigma > 0$ with envelope derivative $\frac{\partial}{\partial\sigma} L_{\lambda_0}(\widehat{\beta}(\sigma), \sigma) = \frac{1-a}{2} - \frac{|y - X\widehat{\beta}(\sigma)|_2^2}{2n\sigma^2}$ (Eq. 6; Proof of Proposition 1, Eqs. A1-A5, pp. 14-15), and the alternating fixed-point equilibrium $(\widehat{\beta}, \widehat{\sigma})$ of Eqs. (3)-(4) satisfies the joint KKT conditions $\widehat{\sigma}^2 = |y - X\widehat{\beta}|_2^2 / \{n(1-a)\}$, $x_j'(y - X\widehat{\beta}) / (n\widehat{\sigma}) = \lambda_0 \operatorname{sign}(\widehat{\beta}_j)$ for $\widehat{\beta}_j \ne 0$, and $|x_j'(y - X\widehat{\beta})| / (n\widehat{\sigma}) \le \lambda_0$ for $\widehat{\beta}_j = 0$ (Proposition 1, Eqs. 7-8, p. 4).
Mechanism
#kkt-residual-sandwich-student-t-tail-bound-and-asymptotic-normality-of-noise-scale
Let $\sigma^* = |y - X\beta^*|_2 / n^{1/2}$ be the oracle noise level for a target vector $\beta^*$ with support $T$. The joint KKT conditions yield the two-sided residual-square sandwich $|y - X\beta^*|_2^2 - n\lambda_0\widehat{\sigma}|\beta^*|_1 \le n(1-a)\widehat{\sigma}^2 \le |y - X\beta^*|_2^2 + n\lambda_0\widehat{\sigma}|\beta^*|_1$ (Appendix, Eq. A6, p. 15). On the scale-free score event $|X'(y - X\beta^*)|_\infty / (n\sigma^*) \le (1-a)^{1/2}\lambda_0 (\xi-1)/(\xi+1)$ ($\xi > 1$), which holds with probability at least $1 - \epsilon$ under $y - X\beta^* \sim N(0, \sigma^2 I_n)$ when $\lambda_0 \ge \frac{\xi+1}{\xi-1}\{(1-a)(1 + 2a_0/n)\}^{-1/2}\{(2/n)\log(p/\epsilon)\}^{1/2}$ via the exact Student-$t$ projection identity $z_j / (|y - X\beta^*|_2/n^{1/2}) = t_j / \{(n - 1 + t_j^2)/n\}^{1/2}$ with $t_j \sim t_{n-1}$ (Lemma 1, Eq. 19, p. 7), Theorem 1 (Eqs. 16-18) and Theorem 2 (Eqs. 22-23, p. 8) prove that with $a = 0$, $\max(1 - \widehat{\sigma}/\sigma^*, 1 - \sigma^*/\widehat{\sigma}) \le \tau_*^2$ where $\tau_*^2 \asymp |T|\lambda_0^2 / \gamma_T = O(|T|(\log p)/n)$, and therefore $n^{1/2}(\widehat{\sigma}/\sigma - 1) = n^{1/2}(\sigma^*/\sigma - 1) + o_P(1) \xrightarrow{d} N(0, 1/2)$ whenever $|\beta|_0(\log p)/n^{1/2} \to 0$ (Theorem 2, Eq. 23; Corollary 2, p. 8).
Mechanism
#cone-invertibility-factor-oracle-bounds-and-post-selection-lse-recovery
Under the sign-restricted cone invertibility factor $F_q(\xi, S) = \inf\{|S|^{1/q}|X'u|_\infty / (n|u|_q) : u \in \mathscr{C}_-(\xi, S)\}$ on $\mathscr{C}_-(\xi, S) = \{u : |u_{S^c}|_1 \le \xi|u_S|_1 \ne 0,\, u_j x_j'X u \le 0\; \forall j \notin S\}$ (Eqs. 10-12, p. 5), Scaled LASSO achieves simultaneous scale-free $\ell_q$ parameter estimation $|\widehat{\beta} - \beta|_q \le (1 + \tau_+)\sigma\{2(\xi+1)|S|^{1/q}\lambda_0 / (\xi F_q(\xi, S))\} = O_P(\sigma|S|^{1/q}\{(\log p)/n\}^{1/2})$ for all $q \in [1, \infty]$ (Theorem 1, Eq. 18; Corollary 1, Eqs. 20-21, pp. 6-7). Furthermore, under the sparse upper eigenvalue $\kappa_+(m, T) = \max_{B \cap T = \emptyset,\, |B| \le m} \lambda_{\max}(X_B'X_B/n)$ with $m > \{2(\xi+1)/(\xi- 1)\}^2 \kappa_+(m, T)|T|/\gamma(\xi, T)$, Scaled LASSO bounds the number of false positives by $|\widehat{S} \setminus S| < m$ (Lemma 2, p. 9), and the refitted Least-Squares Estimator after Model Selection (LSE, $\widetilde{\beta}$, $\widetilde{\sigma}$ on $\widehat{S}$, Eq. 24) satisfies $\max(1 - \widetilde{\sigma}/\sigma^*, 1 - \sigma^*/\widetilde{\sigma}) \le \widetilde{\tau}_* = O_P(|S|(\log p)/n)$ and $|X(\widetilde{\beta} - \beta^*)|_2^2 / n \le 2(\sigma^*)^2 \widetilde{\tau}_* = O_P(\sigma^2 |S|(\log p)/n)$ (Theorem 3, Eqs. 26-27, pp. 9-10).
⚠️ Curated Limitations & Negative Results
- Asymptotic normality $n^{1/2}(\widehat{\sigma}/\sigma - 1) \xrightarrow{d} N(0, 1/2)$ in Theorem 2 and Corollary 2 requires the stronger ultra-sparsity growth condition $|\beta^*|_0(\log p)/n^{1/2} \to 0$ (so that $\tau_*^2 = O(|\beta^*|_0(\log p)/n) = o(n^{-1/2})$) as well as Gaussian errors $y - X\beta^* \sim N(0, \sigma^2 I_n)$; when $|\beta^*|_0(\log p)/n \to 0$ but $|\beta^*|_0(\log p)/n^{1/2} \not\to 0$, $\widehat{\sigma}/\sigma \xrightarrow{P} 1$ Remains consistent at rate $O_P(|\beta^*|_0(\log p)/n)$ but is no longer $n^{1/2}$-negligible relative to $\sigma^*/\sigma - 1$.
- When the alternating equilibrium algorithm (Eqs. 3-4) is used with nonconvex concave penalties $\rho(\beta)$ such as MCP or SCAD instead of $\ell_1$, the joint objective is no longer globally convex and uniqueness of the equilibrium path requires local concavity curvature bounds.
- Primary-source typographical audit of `arXiv:1104.4595v2`: (1) Section 3.3 (p. 8, line 353) defines the sparse upper eigenvalue $\kappa_+(m, T)$ as $\min_{B \cap T \emptyset, |B| \le m} \lambda_{\min}(X_B'X_B/n)$ instead of $\max_{B \cap T = \emptyset, |B| \le m} \lambda_{\max}(X_B'X_B/n)$ (as confirmed by the prose 'Let $\kappa_+(m, T)$ be the sparse upper eigenvalue' on line 351 and its use as an upper bound on $\sum_{j \in B} (x_j'X u / |X u|_2)^2 \le \kappa_+(m, T)$ in the proof of Lemma 2 on p. 18, line 811); (2) Section 4.1 (p. 10, line 446) writes $\widehat{\sigma}^{(\text{bc})} = |y - \widehat{\beta}(\widehat{\lambda}_{\text{cv}})|_2 / (n - |\widehat{S}_{\text{cv}}| - 1)^{1/2}$, omitting the design matrix $X$ before $\widehat{\beta}$; (3) Section 4.2 (p. 12, line 622) prints duplicated word 'at at' ('computed at at a grid'); (4) Appendix Proof of Proposition 1 (p. 14, line 663) writes $L_{\lambda_0}(b, s) = |y - X b|_2^2/(2ns) + (1-a)s/2 + \lambda_0 \sum_{j=1}^p |\beta_j|$ mixing dummy argument $b$ with summand $|\beta_j|$; (5) Appendix Proof of Theorem 1 (p. 16, line 762) defines $c = (|z_T|_\infty / \sigma^*) |w_T|_1 + 2\lambda |w_{T^c}|_1 - (|z_{T^c}|_\infty / \sigma^*) |w_{T^c}|_1$ with $2\lambda$ instead of $\lambda$, while the next display on line 765 correctly uses $(\lambda - |z_{T^c}|_\infty / \sigma^*) |w_{T^c}|_1$; and (6) Appendix Proof of Theorem 3 (p. 18, line 836) omits a closing parenthesis in '$\phi_+ \le \phi(\sigma^*)/(1 - \phi(\sigma^*)$'.
Artificial Intelligence Ontology (AIO) Grounding
Mapped AIO Classes
| AIO Target Class | Mapping Relation | Mapping Rationale |
|---|---|---|
AIO:LassoRegression
Lasso Regression
|
broad_match | Scaled LASSO generalizes LASSO regression by jointly minimizing the Huber concomitant penalized objective over both the sparse regression vector beta and the concomitant noise scale sigma. |
AIO:LinearRegression
Linear Regression
|
broad_match | Estimates both the regression coefficients beta and error standard deviation sigma in high-dimensional linear regression models y = X*beta + epsilon. |
Evidence Sources & Primary Literature
3 Sources| Source ID | Publication / Document | Type / Access | Reference & Checksum |
|---|---|---|---|
sz2012-scaled-lasso-pdf
|
Tingni Sun, Cun-Hui Zhang
|
primary paper
full_text
|
ID: https://doi.org/10.1093/biomet/ass043
sha256:e3be5f1f237f…
|
sz2012-scaled-lasso-abs
|
Tingni Sun, Cun-Hui Zhang
|
primary paper
abstract_only
|
ID: https://arxiv.org/abs/1104.4595
sha256:62de2b085b3d…
|
sz2012-scaled-lasso-crossref
|
T. Sun, C.-H. Zhang
|
primary paper
abstract_only
|
ID: https://doi.org/10.1093/biomet/ass043
sha256:30bb16a6f727…
|
Ontology & Grounding Topology
Open in interactive network graph →Curation Provenance
Curator: Antigravity
Method: ai_assisted
Model: Gemini
Created: 2026-10-19T02:45:00Z
Updated: 2026-10-19T02:45:00Z
Explore across X-Mech
Related computational & biological Mechs:
Embedding Map
Network Graph
ProteinTraitsMech (ESM-2 pLM)
DUFMech (Domain pLM)
TraitMech
CultureMech
All 14 X-Mech Repositories