We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat.ML

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Machine Learning

Title: Free Probability, Newton lilypads and hyperbolicity of Jacobians as a solution to the problem of tuning the architecture of neural networks

Abstract: Gradient descent during the learning process of a neural network can be subject to many instabilities. The spectral density of the Jacobian is a key component for analyzing robustness. Following the works of Pennington et al., such Jacobians are modeled using free multiplicative convolutions from Free Probability Theory (FPT). We present a reliable and very fast method for computing the associated spectral densities. This method has a controlled and proven convergence.
Our technique is based on an homotopy method: it is an adaptative Newton-Raphson scheme which chains basins of attraction. We find contiguous lilypad-like basins and step from one to the next, heading towards the objective.
In order to demonstrate the applicability of our method we show that the relevant FPT metrics computed before training are highly correlated to final test losses -- up to $85 \%$. We also give evidence that a very desirable feature for neural networks is the hyperbolicity of their Jacobian at initialization.
Comments: 22 pages, 4 figures ; v1: Preliminary version ; v2: Added numerical benchmarks, and changed presentation
Subjects: Machine Learning (stat.ML); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Optimization and Control (math.OC); Probability (math.PR)
Cite as: arXiv:2111.00841 [stat.ML]
  (or arXiv:2111.00841v2 [stat.ML] for this version)

Submission history

From: Reda Chhaibi [view email]
[v1] Mon, 1 Nov 2021 11:22:42 GMT (108kb,D)
[v2] Wed, 9 Mar 2022 13:34:00 GMT (175kb,D)

Link back to: arXiv, form interface, contact.