Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
(Submitted on 5 Nov 2017 (v1), last revised 23 Feb 2019 (this version, v2))
Abstract: We study the relationship between geometry and capacity measures for deep neural networks from an invariance viewpoint. We introduce a new notion of capacity --- the Fisher-Rao norm --- that possesses desirable invariance properties and is motivated by Information Geometry. We discover an analytical characterization of the new capacity measure, through which we establish norm-comparison inequalities and further show that the new measure serves as an umbrella for several existing norm-based complexity measures. We discuss upper bounds on the generalization error induced by the proposed measure. Extensive numerical experiments on CIFAR-10 support our theoretical findings. Our theoretical analysis rests on a key structural lemma about partial derivatives of multi-layer rectifier networks.
Submission history
From: James Stokes [view email][v1] Sun, 5 Nov 2017 04:32:59 GMT (212kb,D)
[v2] Sat, 23 Feb 2019 21:27:30 GMT (196kb,D)
Link back to: arXiv, form interface, contact.