Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks
(Submitted on 26 Sep 2019 (v1), last revised 15 Feb 2020 (this version, v4))
Abstract: Recent theoretical work has guaranteed that overparameterized networks trained by gradient descent achieve arbitrarily low training error, and sometimes even low test error. The required width, however, is always polynomial in at least one of the sample size $n$, the (inverse) target error $1/\epsilon$, and the (inverse) failure probability $1/\delta$. This work shows that $\widetilde{\Theta}(1/\epsilon)$ iterations of gradient descent with $\widetilde{\Omega}(1/\epsilon^2)$ training examples on two-layer ReLU networks of any width exceeding $\mathrm{polylog}(n,1/\epsilon,1/\delta)$ suffice to achieve a test misclassification error of $\epsilon$. We also prove that stochastic gradient descent can achieve $\epsilon$ test error with polylogarithmic width and $\widetilde{\Theta}(1/\epsilon)$ samples. The analysis relies upon the separation margin of the limiting kernel, which is guaranteed positive, can distinguish between true labels and random labels, and can give a tight sample-complexity analysis in the infinite-width setting
Submission history
From: Ziwei Ji [view email][v1] Thu, 26 Sep 2019 17:56:28 GMT (18kb)
[v2] Sun, 29 Sep 2019 02:21:50 GMT (18kb)
[v3] Fri, 22 Nov 2019 05:48:27 GMT (24kb)
[v4] Sat, 15 Feb 2020 03:53:09 GMT (24kb)
Link back to: arXiv, form interface, contact.