Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: A Geometric Modeling of Occam's Razor in Deep Learning
(Submitted on 27 May 2019 (v1), last revised 28 Dec 2023 (this version, v5))
Abstract: Why do deep neural networks (DNNs) benefit from very high dimensional parameter spaces? Their huge parameter complexities vs. stunning performances in practice is all the more intriguing and not explainable using the standard theory of regular models. In this work, we propose a geometrically flavored information-theoretic approach to study this phenomenon. Namely, we introduce the locally varying dimensionality of the parameter space of neural network models by considering the number of significant dimensions of the Fisher information matrix, and model the parameter space as a manifold using the framework of singular semi-Riemannian geometry. We derive model complexity measures which yield short description lengths for deep neural network models based on their singularity analysis thus explaining the good performance of DNNs despite their large number of parameters.
Submission history
From: Ke Sun [view email][v1] Mon, 27 May 2019 07:57:26 GMT (1414kb,D)
[v2] Wed, 19 Feb 2020 05:53:28 GMT (122kb,D)
[v3] Wed, 31 Mar 2021 03:45:04 GMT (111kb,D)
[v4] Thu, 23 Dec 2021 12:01:37 GMT (136kb,D)
[v5] Thu, 28 Dec 2023 08:57:54 GMT (54kb,D)
Link back to: arXiv, form interface, contact.