We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Emergent Structures and Lifetime Structure Evolution in Artificial Neural Networks

Abstract: Motivated by the flexibility of biological neural networks whose connectivity structure changes significantly during their lifetime, we introduce the Unstructured Recursive Network (URN) and demonstrate that it can exhibit similar flexibility during training via gradient descent. We show empirically that many of the different neural network structures commonly used in practice today (including fully connected, locally connected and residual networks of different depths and widths) can emerge dynamically from the same URN. These different structures can be derived using gradient descent on a single general loss function where the structure of the data and the relative strengths of various regulator terms determine the structure of the emergent network. We show that this loss function and the regulators arise naturally when considering the symmetries of the network as well as the geometric properties of the input data.
Comments: Proceedings of NeurIPS workshop on Real Neurons & Hidden Units. 5 Pages, 6 figures
Subjects: Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE); Neurons and Cognition (q-bio.NC); Machine Learning (stat.ML)
Cite as: arXiv:1911.11691 [cs.LG]
  (or arXiv:1911.11691v1 [cs.LG] for this version)

Submission history

From: Siavash Golkar [view email]
[v1] Tue, 26 Nov 2019 16:51:37 GMT (703kb,D)

Link back to: arXiv, form interface, contact.