Current browse context:
stat.ML
Change to browse by:
References & Citations
Statistics > Machine Learning
Title: Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations
(Submitted on 9 Aug 2017 (v1), last revised 20 Dec 2017 (this version, v3))
Abstract: This article concerns the expressive power of depth in neural nets with ReLU activations and bounded width. We are particularly interested in the following questions: what is the minimal width $w_{\text{min}}(d)$ so that ReLU nets of width $w_{\text{min}}(d)$ (and arbitrary depth) can approximate any continuous function on the unit cube $[0,1]^d$ aribitrarily well? For ReLU nets near this minimal width, what can one say about the depth necessary to approximate a given function? Our approach to this paper is based on the observation that, due to the convexity of the ReLU activation, ReLU nets are particularly well-suited for representing convex functions. In particular, we prove that ReLU nets with width $d+1$ can approximate any continuous convex function of $d$ variables arbitrarily well. These results then give quantitative depth estimates for the rate of approximation of any continuous scalar function on the $d$-dimensional cube $[0,1]^d$ by ReLU nets with width $d+3.$
Submission history
From: Boris Hanin [view email][v1] Wed, 9 Aug 2017 01:37:21 GMT (18kb)
[v2] Mon, 21 Aug 2017 22:40:55 GMT (20kb)
[v3] Wed, 20 Dec 2017 17:38:26 GMT (11kb)
Link back to: arXiv, form interface, contact.