We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

math-ph

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Mathematical Physics

Title: Universal characteristics of deep neural network loss surfaces from random matrix theory

Abstract: This paper considers several aspects of random matrix universality in deep neural networks. Motivated by recent experimental work, we use universal properties of random matrices related to local statistics to derive practical implications for deep neural networks based on a realistic model of their Hessians. In particular we derive universal aspects of outliers in the spectra of deep neural networks and demonstrate the important role of random matrix local laws in popular pre-conditioning gradient descent algorithms. We also present insights into deep neural network loss surfaces from quite general arguments based on tools from statistical physics and random matrix theory.
Comments: 42 pages
Subjects: Mathematical Physics (math-ph); Disordered Systems and Neural Networks (cond-mat.dis-nn); Machine Learning (cs.LG)
Cite as: arXiv:2205.08601 [math-ph]
  (or arXiv:2205.08601v2 [math-ph] for this version)

Submission history

From: Nicholas Baskerville [view email]
[v1] Tue, 17 May 2022 19:42:23 GMT (280kb,D)
[v2] Mon, 20 Jun 2022 21:34:04 GMT (280kb,D)

Link back to: arXiv, form interface, contact.