We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networks

Abstract: We develop a mathematically rigorous framework for multilayer neural networks in the mean field regime. As the network's widths increase, the network's learning trajectory is shown to be well captured by a meaningful and dynamically nonlinear limit (the \textit{mean field} limit), which is characterized by a system of ODEs. Our framework applies to a broad range of network architectures, learning dynamics and network initializations. Central to the framework is the new idea of a \textit{neuronal embedding}, which comprises of a non-evolving probability space that allows to embed neural networks of arbitrary widths.
Using our framework, we prove several properties of large-width multilayer neural networks. Firstly we show that independent and identically distributed initializations cause strong degeneracy effects on the network's learning trajectory when the network's depth is at least four. Secondly we obtain several global convergence guarantees for feedforward multilayer networks under a number of different setups. These include two-layer and three-layer networks with independent and identically distributed initializations, and multilayer networks of arbitrary depths with a special type of correlated initializations that is motivated by the new concept of \textit{bidirectional diversity}. Unlike previous works that rely on convexity, our results admit non-convex losses and hinge on a certain universal approximation property, which is a distinctive feature of infinite-width neural networks and is shown to hold throughout the training process. Aside from being the first known results for global convergence of multilayer networks in the mean field regime, they demonstrate flexibility of our framework and incorporate several new ideas and insights that depart from the conventional convex optimization wisdom.
Comments: 116 pages. This version incorporates the content of the companion note arXiv:2006.09355 (June 2020)
Subjects: Machine Learning (cs.LG); Statistical Mechanics (cond-mat.stat-mech); Probability (math.PR); Statistics Theory (math.ST); Machine Learning (stat.ML)
Cite as: arXiv:2001.11443 [cs.LG]
  (or arXiv:2001.11443v2 [cs.LG] for this version)

Submission history

From: Huy Tuan Pham [view email]
[v1] Thu, 30 Jan 2020 16:43:34 GMT (49kb)
[v2] Tue, 4 May 2021 17:44:02 GMT (90kb)

Link back to: arXiv, form interface, contact.