Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Improved multitask learning through synaptic intelligence
(Submitted on 13 Mar 2017 (v1), revised 10 Apr 2017 (this version, v2), latest version 12 Jun 2017 (v3))
Abstract: Deep learning has led to remarkable advances when applied to problems where the data distribution does not change over the course of learning. In stark contrast, biological neural networks continually adapt to changing domains, and solve a diversity of tasks simultaneously. Furthermore, synapses in biological neurons are not simply real-valued scalars, but possess complex molecular machinery enabling non-trivial learning dynamics. In this study, we take a first step toward bringing this biological complexity into artificial neural networks. We introduce a model of intelligent synapses that accumulate task relevant information over time, and exploit this information to efficiently consolidate memories of old tasks to protect them from being overwritten as new tasks are learned. We apply our framework to learning sequences of related classification problems, and show that it dramatically reduces catastrophic forgetting while maintaining computational efficiency.
Submission history
From: Ben Poole [view email][v1] Mon, 13 Mar 2017 00:02:48 GMT (255kb,D)
[v2] Mon, 10 Apr 2017 17:54:57 GMT (259kb,D)
[v3] Mon, 12 Jun 2017 19:57:42 GMT (1060kb,D)
Link back to: arXiv, form interface, contact.