We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cond-mat.dis-nn

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Condensed Matter > Disordered Systems and Neural Networks

Title: Boundary between noise and information applied to filtering neural network weight matrices

Abstract: Deep neural networks have been successfully applied to a broad range of problems where overparametrization yields weight matrices which are partially random. A comparison of weight matrix singular vectors to the Porter-Thomas distribution suggests that there is a boundary between randomness and learned information in the singular value spectrum. Inspired by this finding, we introduce an algorithm for noise filtering, which both removes small singular values and reduces the magnitude of large singular values to counteract the effect of level repulsion between the noise and the information part of the spectrum. For networks trained in the presence of label noise, we indeed find that the generalization performance improves significantly due to noise filtering.
Comments: 6 pages, 5 figures
Subjects: Disordered Systems and Neural Networks (cond-mat.dis-nn); Machine Learning (cs.LG)
Cite as: arXiv:2206.03927 [cond-mat.dis-nn]
  (or arXiv:2206.03927v1 [cond-mat.dis-nn] for this version)

Submission history

From: Matthias Thamm [view email]
[v1] Wed, 8 Jun 2022 14:42:36 GMT (675kb,D)

Link back to: arXiv, form interface, contact.