We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat.ML

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Machine Learning

Title: Towards Optimality Conditions for Non-Linear Networks

Abstract: Training non-linear neural networks is a challenging task, but over the years, various approaches coming from different perspectives have been proposed to improve performance. However, insights into what fundamentally constitutes \textit{optimal} network parameters remains obscure. Similarly, given what properties of data can we hope for a non-linear network to learn is also not well studied. In order to address these challenges, we take a novel approach by analysing neural network from a data generating perspective, where we assume hidden layers generate the observed data. This perspective allows us to connect seemingly disparate approaches explored independently in the machine learning community such as batch normalization, Independent Component Analysis, orthogonal weight initialization, etc, as parts of a bigger picture and provide insights into non-linear networks in terms of properties of parameter and data that lead to better performance.
Comments: 10 pages of main content, 8 pages of appendix
Subjects: Machine Learning (stat.ML); Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE)
Cite as: arXiv:1605.07145 [stat.ML]
  (or arXiv:1605.07145v1 [stat.ML] for this version)

Submission history

From: Devansh Arpit [view email]
[v1] Mon, 23 May 2016 19:21:53 GMT (822kb,D)
[v2] Thu, 13 Jul 2017 16:25:15 GMT (373kb,D)

Link back to: arXiv, form interface, contact.