We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

math.ST

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Mathematics > Statistics Theory

Title: Distribution-Free Robust Linear Regression

Abstract: We study random design linear regression with no assumptions on the distribution of the covariates and with a heavy-tailed response variable. In this distribution-free regression setting, we show that boundedness of the conditional second moment of the response given the covariates is a necessary and sufficient condition for achieving nontrivial guarantees. As a starting point, we prove an optimal version of the classical in-expectation bound for the truncated least squares estimator due to Gy\"{o}rfi, Kohler, Krzy\.{z}ak, and Walk. However, we show that this procedure fails with constant probability for some distributions despite its optimal in-expectation performance. Then, combining the ideas of truncated least squares, median-of-means procedures, and aggregation theory, we construct a non-linear estimator achieving excess risk of order $d/n$ with an optimal sub-exponential tail. While existing approaches to linear regression for heavy-tailed distributions focus on proper estimators that return linear functions, we highlight that the improperness of our procedure is necessary for attaining nontrivial guarantees in the distribution-free setting.
Comments: 29 pages, to appear in Mathematical Statistics and Learning
Subjects: Statistics Theory (math.ST); Machine Learning (cs.LG); Machine Learning (stat.ML)
Journal reference: Math. Stat. Learn. 4 (2021), 253-292
DOI: 10.4171/MSL/27
Cite as: arXiv:2102.12919 [math.ST]
  (or arXiv:2102.12919v2 [math.ST] for this version)

Submission history

From: Nikita Zhivotovskiy [view email]
[v1] Thu, 25 Feb 2021 15:10:41 GMT (35kb)
[v2] Thu, 21 Oct 2021 10:42:49 GMT (36kb)

Link back to: arXiv, form interface, contact.