### Current browse context:

stat.ML

### Change to browse by:

### References & Citations

# Statistics > Machine Learning

# Title: Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev Spaces

(Submitted on 25 Nov 2022 (v1), last revised 6 Jan 2023 (this version, v2))

Abstract: Let $\Omega = [0,1]^d$ be the unit cube in $\mathbb{R}^d$. We study the problem of how efficiently, in terms of the number of parameters, deep neural networks with the ReLU activation function can approximate functions in the Sobolev space $W^s(L_q(\Omega))$ with error measured in $L_p(\Omega)$. This problem is important when studying the application of neural networks in scientific computing and has previously been completely solved only in the case $p=q=\infty$. Our contribution is to provide a complete solution for all $1\leq p,q\leq \infty$ and $s > 0$, including asymptotically matching upper and lower bounds. The key technical tool is a novel bit-extraction technique which gives an optimal encoding of sparse vectors. This enables us to obtain sharp upper bounds in the non-linear regime where $p > q$. We also provide a novel method for deriving $L_p$-approximation lower bounds based upon VC-dimension when $p < \infty$. Our results show that very deep ReLU networks significantly outperform classical methods of approximation in terms of the number of parameters, but that this comes at the cost of parameters which are not encodable.

## Submission history

From: Jonathan Siegel [view email]**[v1]**Fri, 25 Nov 2022 23:32:26 GMT (35kb)

**[v2]**Fri, 6 Jan 2023 07:20:23 GMT (39kb)

Link back to: arXiv, form interface, contact.