We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Machine Learning

Title: Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigm

Abstract: How many training data are needed to learn a supervised task? It is often observed that the generalization error decreases as $n^{-\beta}$ where $n$ is the number of training examples and $\beta$ an exponent that depends on both data and algorithm. In this work we measure $\beta$ when applying kernel methods to real datasets. For MNIST we find $\beta\approx 0.4$ and for CIFAR10 $\beta\approx 0.1$, for both regression and classification tasks, and for Gaussian or Laplace kernels. To rationalize the existence of non-trivial exponents that can be independent of the specific kernel used, we study the Teacher-Student framework for kernels. In this scheme, a Teacher generates data according to a Gaussian random field, and a Student learns them via kernel regression. With a simplifying assumption -- namely that the data are sampled from a regular lattice -- we derive analytically $\beta$ for translation invariant kernels, using previous results from the kriging literature. Provided that the Student is not too sensitive to high frequencies, $\beta$ depends only on the smoothness and dimension of the training data. We confirm numerically that these predictions hold when the training points are sampled at random on a hypersphere. Overall, the test error is found to be controlled by the magnitude of the projection of the true function on the kernel eigenvectors whose rank is larger than $n$. Using this idea we predict relate the exponent $\beta$ to an exponent $a$ describing how the coefficients of the true function in the eigenbasis of the kernel decay with rank. We extract $a$ from real data by performing kernel PCA, leading to $\beta\approx0.36$ for MNIST and $\beta\approx0.07$ for CIFAR10, in good agreement with observations. We argue that these rather large exponents are possible due to the small effective dimension of the data.
Comments: We added (i) the prediction of the exponent $\beta$ for real data using kernel PCA; (ii) the generalization of our results to non-Gaussian data from reference [11] (Bordelon et al., "Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks")
Subjects: Machine Learning (stat.ML); Disordered Systems and Neural Networks (cond-mat.dis-nn); Machine Learning (cs.LG)
DOI: 10.1088/1742-5468/abc61d
Cite as: arXiv:1905.10843 [stat.ML]
  (or arXiv:1905.10843v8 [stat.ML] for this version)

Submission history

From: Stefano Spigler [view email]
[v1] Sun, 26 May 2019 17:29:11 GMT (337kb,D)
[v2] Tue, 28 May 2019 07:55:56 GMT (337kb,D)
[v3] Wed, 29 May 2019 11:50:27 GMT (337kb,D)
[v4] Thu, 6 Jun 2019 14:11:28 GMT (337kb,D)
[v5] Wed, 24 Jul 2019 08:11:22 GMT (337kb,D)
[v6] Thu, 6 Aug 2020 09:18:25 GMT (512kb,D)
[v7] Thu, 13 Aug 2020 10:05:46 GMT (512kb,D)
[v8] Tue, 18 Aug 2020 13:27:19 GMT (512kb,D)

Link back to: arXiv, form interface, contact.