We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat.ML

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Machine Learning

Title: Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev Spaces

Abstract: Let $\Omega = [0,1]^d$ be the unit cube in $\mathbb{R}^d$. We study the problem of how efficiently, in terms of the number of parameters, deep neural networks with the ReLU activation function can approximate functions in the Sobolev space $W^s(L_q(\Omega))$ with error measured in $L_p(\Omega)$. This problem is important when studying the application of neural networks in scientific computing and has previously been completely solved only in the case $p=q=\infty$. Our contribution is to provide a complete solution for all $1\leq p,q\leq \infty$ and $s > 0$, including asymptotically matching upper and lower bounds. The key technical tool is a novel bit-extraction technique which gives an optimal encoding of sparse vectors. This enables us to obtain sharp upper bounds in the non-linear regime where $p > q$. We also provide a novel method for deriving $L_p$-approximation lower bounds based upon VC-dimension when $p < \infty$. Our results show that very deep ReLU networks significantly outperform classical methods of approximation in terms of the number of parameters, but that this comes at the cost of parameters which are not encodable.
Subjects: Machine Learning (stat.ML); Machine Learning (cs.LG); Numerical Analysis (math.NA)
MSC classes: 41A25, 41A46, 62M45
Cite as: arXiv:2211.14400 [stat.ML]
  (or arXiv:2211.14400v2 [stat.ML] for this version)

Submission history

From: Jonathan Siegel [view email]
[v1] Fri, 25 Nov 2022 23:32:26 GMT (35kb)
[v2] Fri, 6 Jan 2023 07:20:23 GMT (39kb)

Link back to: arXiv, form interface, contact.