### Current browse context:

cs.LG

### Change to browse by:

### References & Citations

# Computer Science > Machine Learning

# Title: Optimal Approximation Rate of ReLU Networks in terms of Width and Depth

(Submitted on 28 Feb 2021 (v1), last revised 24 Jul 2021 (this version, v5))

Abstract: This paper concentrates on the approximation power of deep feed-forward neural networks in terms of width and depth. It is proved by construction that ReLU networks with width $\mathcal{O}\big(\max\{d\lfloor N^{1/d}\rfloor,\, N+2\}\big)$ and depth $\mathcal{O}(L)$ can approximate a H\"older continuous function on $[0,1]^d$ with an approximation rate $\mathcal{O}\big(\lambda\sqrt{d} (N^2L^2\ln N)^{-\alpha/d}\big)$, where $\alpha\in (0,1]$ and $\lambda>0$ are H\"older order and constant, respectively. Such a rate is optimal up to a constant in terms of width and depth separately, while existing results are only nearly optimal without the logarithmic factor in the approximation rate. More generally, for an arbitrary continuous function $f$ on $[0,1]^d$, the approximation rate becomes $\mathcal{O}\big(\,\sqrt{d}\,\omega_f\big( (N^2L^2\ln N)^{-1/d}\big)\,\big)$, where $\omega_f(\cdot)$ is the modulus of continuity. We also extend our analysis to any continuous function $f$ on a bounded set. Particularly, if ReLU networks with depth $31$ and width $\mathcal{O}(N)$ are used to approximate one-dimensional Lipschitz continuous functions on $[0,1]$ with a Lipschitz constant $\lambda>0$, the approximation rate in terms of the total number of parameters, $W=\mathcal{O}(N^2)$, becomes $\mathcal{O}(\tfrac{\lambda}{W\ln W})$, which has not been discovered in the literature for fixed-depth ReLU networks.

## Submission history

From: Shijun Zhang [view email]**[v1]**Sun, 28 Feb 2021 13:15:55 GMT (1592kb,D)

**[v2]**Wed, 24 Mar 2021 08:06:23 GMT (1724kb,D)

**[v3]**Fri, 9 Apr 2021 16:33:29 GMT (1661kb,D)

**[v4]**Sat, 17 Jul 2021 04:05:16 GMT (1855kb,D)

**[v5]**Sat, 24 Jul 2021 16:48:34 GMT (1855kb,D)

Link back to: arXiv, form interface, contact.