We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cond-mat.dis-nn

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Condensed Matter > Disordered Systems and Neural Networks

Title: Theory of gating in recurrent neural networks

Abstract: Recurrent neural networks (RNNs) are powerful dynamical models, widely used in machine learning (ML) and neuroscience. Prior theoretical work has focused on RNNs with additive interactions. However, gating - i.e. multiplicative - interactions are ubiquitous in real neurons and also the central feature of the best-performing RNNs in ML. Here, we show that gating offers flexible control of two salient features of the collective dynamics: i) timescales and ii) dimensionality. The gate controlling timescales leads to a novel, marginally stable state, where the network functions as a flexible integrator. Unlike previous approaches, gating permits this important function without parameter fine-tuning or special symmetries. Gates also provide a flexible, context-dependent mechanism to reset the memory trace, thus complementing the memory function. The gate modulating the dimensionality can induce a novel, discontinuous chaotic transition, where inputs push a stable system to strong chaotic activity, in contrast to the typically stabilizing effect of inputs. At this transition, unlike additive RNNs, the proliferation of critical points (topological complexity) is decoupled from the appearance of chaotic dynamics (dynamical complexity).
The rich dynamics are summarized in phase diagrams, thus providing a map for principled parameter initialization choices to ML practitioners.
Comments: 13 figures
Subjects: Disordered Systems and Neural Networks (cond-mat.dis-nn); Statistical Mechanics (cond-mat.stat-mech); Machine Learning (cs.LG); Chaotic Dynamics (nlin.CD); Neurons and Cognition (q-bio.NC)
Cite as: arXiv:2007.14823 [cond-mat.dis-nn]
  (or arXiv:2007.14823v5 [cond-mat.dis-nn] for this version)

Submission history

From: Kamesh Krishnamurthy [view email]
[v1] Wed, 29 Jul 2020 13:20:58 GMT (5249kb,D)
[v2] Sat, 29 Aug 2020 21:48:52 GMT (5301kb,D)
[v3] Thu, 3 Sep 2020 20:02:16 GMT (5250kb,D)
[v4] Thu, 21 Jan 2021 03:03:56 GMT (5779kb,D)
[v5] Wed, 1 Dec 2021 17:43:29 GMT (4292kb,D)

Link back to: arXiv, form interface, contact.