We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

math.ST

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Mathematics > Statistics Theory

Title: Out-of-sample error estimate for robust M-estimators with convex penalty

Abstract: A generic out-of-sample error estimate is proposed for robust $M$-estimators regularized with a convex penalty in high-dimensional linear regression where $(X,y)$ is observed and $p,n$ are of the same order. If $\psi$ is the derivative of the robust data-fitting loss $\rho$, the estimate depends on the observed data only through the quantities $\hat\psi = \psi(y-X\hat\beta)$, $X^\top \hat\psi$ and the derivatives $(\partial/\partial y) \hat\psi$ and $(\partial/\partial y) X\hat\beta$ for fixed $X$.
The out-of-sample error estimate enjoys a relative error of order $n^{-1/2}$ in a linear model with Gaussian covariates and independent noise, either non-asymptotically when $p/n\le \gamma$ or asymptotically in the high-dimensional asymptotic regime $p/n\to\gamma'\in(0,\infty)$. General differentiable loss functions $\rho$ are allowed provided that $\psi=\rho'$ is 1-Lipschitz. The validity of the out-of-sample error estimate holds either under a strong convexity assumption, or for the $\ell_1$-penalized Huber M-estimator if the number of corrupted observations and sparsity of the true $\beta$ are bounded from above by $s_*n$ for some small enough constant $s_*\in(0,1)$ independent of $n,p$.
For the square loss and in the absence of corruption in the response, the results additionally yield $n^{-1/2}$-consistent estimates of the noise variance and of the generalization error. This generalizes, to arbitrary convex penalty, estimates that were previously known for the Lasso.
Comments: This version adds simulations for the nuclear norm penalty
Subjects: Statistics Theory (math.ST); Machine Learning (stat.ML)
Cite as: arXiv:2008.11840 [math.ST]
  (or arXiv:2008.11840v4 [math.ST] for this version)

Submission history

From: Pierre C. Bellec [view email]
[v1] Wed, 26 Aug 2020 21:50:41 GMT (61kb)
[v2] Mon, 14 Sep 2020 23:23:24 GMT (64kb)
[v3] Mon, 12 Oct 2020 14:48:21 GMT (160kb,D)
[v4] Thu, 3 Jun 2021 20:17:13 GMT (191kb,D)

Link back to: arXiv, form interface, contact.