We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

nlin.AO

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Nonlinear Sciences > Adaptation and Self-Organizing Systems

Title: Emergence of Network Motifs in Deep Neural Networks

Abstract: Network science can offer fundamental insights into the structural and functional properties of complex systems. For example, it is widely known that neuronal circuits tend to organize into basic functional topological modules, called "network motifs". In this article we show that network science tools can be successfully applied also to the study of artificial neural networks operating according to self-organizing (learning) principles. In particular, we study the emergence of network motifs in multi-layer perceptrons, whose initial connectivity is defined as a stack of fully-connected, bipartite graphs. Our simulations show that the final network topology is primarily shaped by learning dynamics, but can be strongly biased by choosing appropriate weight initialization schemes. Overall, our results suggest that non-trivial initialization strategies can make learning more effective by promoting the development of useful network motifs, which are often surprisingly consistent with those observed in general transduction networks.
Subjects: Adaptation and Self-Organizing Systems (nlin.AO); Machine Learning (cs.LG); Biological Physics (physics.bio-ph)
DOI: 10.3390/e22020204
Cite as: arXiv:1912.12244 [nlin.AO]
  (or arXiv:1912.12244v1 [nlin.AO] for this version)

Submission history

From: Alberto Testolin Dr. [view email]
[v1] Fri, 27 Dec 2019 17:05:38 GMT (4840kb,D)

Link back to: arXiv, form interface, contact.