We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: On the Generalization Power of Overfitted Two-Layer Neural Tangent Kernel Models

Abstract: In this paper, we study the generalization performance of min $\ell_2$-norm overfitting solutions for the neural tangent kernel (NTK) model of a two-layer neural network with ReLU activation that has no bias term. We show that, depending on the ground-truth function, the test error of overfitted NTK models exhibits characteristics that are different from the "double-descent" of other overparameterized linear models with simple Fourier or Gaussian features. Specifically, for a class of learnable functions, we provide a new upper bound of the generalization error that approaches a small limiting value, even when the number of neurons $p$ approaches infinity. This limiting value further decreases with the number of training samples $n$. For functions outside of this class, we provide a lower bound on the generalization error that does not diminish to zero even when $n$ and $p$ are both large.
Comments: Published in ICML21. This version fixes an error of Lemma 31 and other parts affected by this error. The main results remain the same except some small changes on certain coefficients of Eq.(9)
Subjects: Machine Learning (cs.LG); Statistics Theory (math.ST); Machine Learning (stat.ML)
Cite as: arXiv:2103.05243 [cs.LG]
  (or arXiv:2103.05243v3 [cs.LG] for this version)

Submission history

From: Peizhong Ju [view email]
[v1] Tue, 9 Mar 2021 06:24:59 GMT (457kb)
[v2] Tue, 24 Aug 2021 05:18:52 GMT (607kb)
[v3] Tue, 7 Mar 2023 20:54:33 GMT (621kb)

Link back to: arXiv, form interface, contact.