Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: An Information-Theoretic Framework for Supervised Learning
(Submitted on 1 Mar 2022 (v1), last revised 24 Mar 2023 (this version, v6))
Abstract: Each year, deep learning demonstrates new and improved empirical results with deeper and wider neural networks. Meanwhile, with existing theoretical frameworks, it is difficult to analyze networks deeper than two layers without resorting to counting parameters or encountering sample complexity bounds that are exponential in depth. Perhaps it may be fruitful to try to analyze modern machine learning under a different lens. In this paper, we propose a novel information-theoretic framework with its own notions of regret and sample complexity for analyzing the data requirements of machine learning. With our framework, we first work through some classical examples such as scalar estimation and linear regression to build intuition and introduce general techniques. Then, we use the framework to study the sample complexity of learning from data generated by deep neural networks with ReLU activation units. For a particular prior distribution on weights, we establish sample complexity bounds that are simultaneously width independent and linear in depth. This prior distribution gives rise to high-dimensional latent representations that, with high probability, admit reasonably accurate low-dimensional approximations. We conclude by corroborating our theoretical results with experimental analysis of random single-hidden-layer neural networks.
Submission history
From: Hong Jun Jeon [view email][v1] Tue, 1 Mar 2022 05:58:28 GMT (141kb,D)
[v2] Fri, 4 Mar 2022 23:53:39 GMT (141kb,D)
[v3] Thu, 7 Apr 2022 22:33:12 GMT (141kb,D)
[v4] Sun, 22 May 2022 20:29:40 GMT (278kb,D)
[v5] Tue, 7 Jun 2022 21:13:39 GMT (932kb,D)
[v6] Fri, 24 Mar 2023 19:48:25 GMT (20342kb,D)
Link back to: arXiv, form interface, contact.