We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat.ML

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Machine Learning

Title: A Model of Double Descent for High-dimensional Binary Linear Classification

Abstract: We consider a model for logistic regression where only a subset of features of size $p$ is used for training a linear classifier over $n$ training samples. The classifier is obtained by running gradient descent (GD) on logistic loss. For this model, we investigate the dependence of the classification error on the overparameterization ratio $\kappa=p/n$. First, building on known deterministic results on the implicit bias of GD, we uncover a phase-transition phenomenon for the case of Gaussian features: the classification error of GD is the same as that of the maximum-likelihood (ML) solution when $\kappa<\kappa_\star$, and that of the max-margin (SVM) solution when $\kappa>\kappa_\star$. Next, using the convex Gaussian min-max theorem (CGMT), we sharply characterize the performance of both the ML and the SVM solutions. Combining these results, we obtain curves that explicitly characterize the classification error for varying values of $\kappa$. The numerical results validate the theoretical predictions and unveil double-descent phenomena that complement similar recent findings in linear regression settings as well as empirical observations in more complex learning scenarios.
Comments: Short version submitted to ICASSP 2020; Updates in 2nd version: revised proofs, typos fixed, extended discussions and numerical illustrations
Subjects: Machine Learning (stat.ML); Machine Learning (cs.LG); Signal Processing (eess.SP)
Cite as: arXiv:1911.05822 [stat.ML]
  (or arXiv:1911.05822v2 [stat.ML] for this version)

Submission history

From: Christos Thrampoulidis [view email]
[v1] Wed, 13 Nov 2019 21:41:38 GMT (272kb)
[v2] Sun, 10 May 2020 19:10:56 GMT (372kb)

Link back to: arXiv, form interface, contact.