We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cond-mat.dis-nn

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Condensed Matter > Disordered Systems and Neural Networks

Title: Unveiling the structure of wide flat minima in neural networks

Abstract: The success of deep learning has revealed the application potential of neural networks across the sciences and opened up fundamental theoretical problems. In particular, the fact that learning algorithms based on simple variants of gradient methods are able to find near-optimal minima of highly nonconvex loss functions is an unexpected feature of neural networks. Moreover, such algorithms are able to fit the data even in the presence of noise, and yet they have excellent predictive capabilities. Several empirical results have shown a reproducible correlation between the so-called flatness of the minima achieved by the algorithms and the generalization performance. At the same time, statistical physics results have shown that in nonconvex networks a multitude of narrow minima may coexist with a much smaller number of wide flat minima, which generalize well. Here we show that wide flat minima arise as complex extensive structures, from the coalescence of minima around "high-margin" (i.e., locally robust) configurations. Despite being exponentially rare compared to zero-margin ones, high-margin minima tend to concentrate in particular regions. These minima are in turn surrounded by other solutions of smaller and smaller margin, leading to dense regions of solutions over long distances. Our analysis also provides an alternative analytical method for estimating when flat minima appear and when algorithms begin to find solutions, as the number of model parameters varies.
Comments: 15 pages, 8 figures
Subjects: Disordered Systems and Neural Networks (cond-mat.dis-nn); Machine Learning (cs.LG); Mathematical Physics (math-ph); Probability (math.PR)
DOI: 10.1103/PhysRevLett.127.278301
Cite as: arXiv:2107.01163 [cond-mat.dis-nn]
  (or arXiv:2107.01163v5 [cond-mat.dis-nn] for this version)

Submission history

From: Enrico Maria Malatesta [view email]
[v1] Fri, 2 Jul 2021 16:04:57 GMT (516kb,D)
[v2] Tue, 26 Oct 2021 15:19:24 GMT (918kb,D)
[v3] Wed, 27 Oct 2021 16:50:28 GMT (934kb,D)
[v4] Tue, 7 Dec 2021 15:41:38 GMT (910kb,D)
[v5] Mon, 14 Feb 2022 10:17:55 GMT (910kb,D)

Link back to: arXiv, form interface, contact.