We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat.ML

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Machine Learning

Title: On the Expected Complexity of Maxout Networks

Abstract: Learning with neural networks relies on the complexity of the representable functions, but more importantly, the particular assignment of typical parameters to functions of different complexity. Taking the number of activation regions as a complexity measure, recent works have shown that the practical complexity of deep ReLU networks is often far from the theoretical maximum. In this work, we show that this phenomenon also occurs in networks with maxout (multi-argument) activation functions and when considering the decision boundaries in classification tasks. We also show that the parameter space has a multitude of full-dimensional regions with widely different complexity, and obtain nontrivial lower bounds on the expected complexity. Finally, we investigate different parameter initialization procedures and show that they can increase the speed of convergence in training.
Comments: Published at NeurIPS 2021, 47 pages, 18 figures
Subjects: Machine Learning (stat.ML); Machine Learning (cs.LG)
Cite as: arXiv:2107.00379 [stat.ML]
  (or arXiv:2107.00379v2 [stat.ML] for this version)

Submission history

From: Hanna Tseran [view email]
[v1] Thu, 1 Jul 2021 11:36:32 GMT (12283kb,D)
[v2] Thu, 16 Dec 2021 08:28:02 GMT (12056kb,D)

Link back to: arXiv, form interface, contact.