We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat.ML

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Machine Learning

Title: A spectral-based analysis of the separation between two-layer neural networks and linear methods

Authors: Lei Wu, Jihao Long
Abstract: We propose a spectral-based approach to analyze how two-layer neural networks separate from linear methods in terms of approximating high-dimensional functions. We show that quantifying this separation can be reduced to estimating the Kolmogorov width of two-layer neural networks, and the latter can be further characterized by using the spectrum of an associated kernel. Different from previous work, our approach allows obtaining upper bounds, lower bounds, and identifying explicit hard functions in a united manner. We provide a systematic study of how the choice of activation functions affects the separation, in particular the dependence on the input dimension. Specifically, for nonsmooth activation functions, we extend known results to more activation functions with sharper bounds. As concrete examples, we prove that any single neuron can instantiate the separation between neural networks and random feature models. For smooth activation functions, one surprising finding is that the separation is negligible unless the norms of inner-layer weights are polynomially large with respect to the input dimension. By contrast, the separation for nonsmooth activation functions is independent of the norms of inner-layer weights.
Comments: Accepted by Journal of Machine Learning Research
Subjects: Machine Learning (stat.ML); Machine Learning (cs.LG)
Cite as: arXiv:2108.04964 [stat.ML]
  (or arXiv:2108.04964v2 [stat.ML] for this version)

Submission history

From: Lei Wu [view email]
[v1] Tue, 10 Aug 2021 23:30:29 GMT (107kb,D)
[v2] Wed, 23 Feb 2022 06:48:01 GMT (129kb,D)

Link back to: arXiv, form interface, contact.