We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers

Abstract: The fundamental learning theory behind neural networks remains largely open. What classes of functions can neural networks actually learn? Why doesn't the trained network overfit when it is overparameterized?
In this work, we prove that overparameterized neural networks can learn some notable concept classes, including two and three-layer networks with fewer parameters and smooth activations. Moreover, the learning can be simply done by SGD (stochastic gradient descent) or its variants in polynomial time using polynomially many samples. The sample complexity can also be almost independent of the number of parameters in the network.
On the technique side, our analysis goes beyond the so-called NTK (neural tangent kernel) linearization of neural networks in prior works. We establish a new notion of quadratic approximation of the neural network (that can be viewed as a second-order variant of NTK), and connect it to the SGD theory of escaping saddle points.
Comments: V1/V2/V3/V4 polish writing, V5 adds experiments, V6 reflects our camera ready version
Subjects: Machine Learning (cs.LG); Data Structures and Algorithms (cs.DS); Neural and Evolutionary Computing (cs.NE); Optimization and Control (math.OC); Machine Learning (stat.ML)
Cite as: arXiv:1811.04918 [cs.LG]
  (or arXiv:1811.04918v6 [cs.LG] for this version)

Submission history

From: Zeyuan Allen-Zhu [view email]
[v1] Mon, 12 Nov 2018 18:57:02 GMT (76kb)
[v2] Thu, 17 Jan 2019 15:56:01 GMT (114kb)
[v3] Mon, 4 Feb 2019 04:10:51 GMT (122kb)
[v4] Fri, 29 Mar 2019 17:09:46 GMT (125kb)
[v5] Tue, 28 May 2019 10:25:09 GMT (507kb,D)
[v6] Mon, 1 Jun 2020 17:11:51 GMT (510kb,D)

Link back to: arXiv, form interface, contact.