We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Effect of Depth and Width on Local Minima in Deep Learning

Abstract: In this paper, we analyze the effects of depth and width on the quality of local minima, without strong over-parameterization and simplification assumptions in the literature. Without any simplification assumption, for deep nonlinear neural networks with the squared loss, we theoretically show that the quality of local minima tends to improve towards the global minimum value as depth and width increase. Furthermore, with a locally-induced structure on deep nonlinear neural networks, the values of local minima of neural networks are theoretically proven to be no worse than the globally optimal values of corresponding classical machine learning models. We empirically support our theoretical observation with a synthetic dataset as well as MNIST, CIFAR-10 and SVHN datasets. When compared to previous studies with strong over-parameterization assumptions, the results in this paper do not require over-parameterization, and instead show the gradual effects of over-parameterization as consequences of general results.
Subjects: Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE); Optimization and Control (math.OC); Machine Learning (stat.ML)
Journal reference: Neural computation, volume 31, pages 1462-1498 (2019)
DOI: 10.1162/neco_a_01195
Cite as: arXiv:1811.08150 [cs.LG]
  (or arXiv:1811.08150v4 [cs.LG] for this version)

Submission history

From: Kenji Kawaguchi [view email]
[v1] Tue, 20 Nov 2018 09:41:52 GMT (417kb,D)
[v2] Mon, 4 Mar 2019 19:48:28 GMT (165kb,D)
[v3] Mon, 17 Jun 2019 18:40:43 GMT (165kb,D)
[v4] Tue, 9 Jul 2019 15:32:37 GMT (537kb)

Link back to: arXiv, form interface, contact.