Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks
(Submitted on 7 Feb 2020 (v1), last revised 25 Feb 2021 (this version, v7))
Abstract: We derive analytical expressions for the generalization performance of kernel regression as a function of the number of training samples using theoretical methods from Gaussian processes and statistical physics. Our expressions apply to wide neural networks due to an equivalence between training them and kernel regression with the Neural Tangent Kernel (NTK). By computing the decomposition of the total generalization error due to different spectral components of the kernel, we identify a new spectral principle: as the size of the training set grows, kernel machines and neural networks fit successively higher spectral modes of the target function. When data are sampled from a uniform distribution on a high-dimensional hypersphere, dot product kernels, including NTK, exhibit learning stages where different frequency modes of the target function are learned. We verify our theory with simulations on synthetic data and MNIST dataset.
Submission history
From: Blake Bordelon [view email][v1] Fri, 7 Feb 2020 00:03:40 GMT (519kb,D)
[v2] Wed, 19 Feb 2020 02:38:09 GMT (523kb,D)
[v3] Sun, 3 May 2020 21:32:34 GMT (621kb,D)
[v4] Sun, 28 Jun 2020 00:06:11 GMT (637kb,D)
[v5] Thu, 13 Aug 2020 21:05:27 GMT (701kb,D)
[v6] Thu, 27 Aug 2020 17:13:23 GMT (639kb,D)
[v7] Thu, 25 Feb 2021 18:40:10 GMT (638kb,D)
Link back to: arXiv, form interface, contact.