We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Encoders and Ensembles for Task-Free Continual Learning

Abstract: We present an architecture that is effective for continual learning in an especially demanding setting, where task boundaries do not exist or are unknown, and where classes have to be learned online (with each example presented only once). To obtain good performance under these constraints, while mitigating catastrophic forgetting, we exploit recent advances in contrastive, self-supervised learning, allowing us to use a pre-trained, general purpose image encoder whose weights can be frozen, which precludes forgetting. The pre-trained encoder also greatly simplifies the downstream task of classification, which we solve with an ensemble of very simple classifiers. Collectively, the ensemble exhibits much better performance than any individual classifier, an effect which is amplified through specialisation and competitive selection. We assess the performance of the encoders-and-ensembles architecture on standard continual learning benchmarks, where it outperforms prior state-of-the-art by a large margin on the hardest problems, as well as in less familiar settings where the data distribution changes gradually or the classes are presented one at a time.
Subjects: Machine Learning (cs.LG)
Cite as: arXiv:2105.13327 [cs.LG]
  (or arXiv:2105.13327v5 [cs.LG] for this version)

Submission history

From: Murray Shanahan [view email]
[v1] Thu, 27 May 2021 17:34:31 GMT (2674kb,D)
[v2] Thu, 15 Jul 2021 15:07:13 GMT (1271kb,D)
[v3] Mon, 19 Jul 2021 10:30:31 GMT (1272kb,D)
[v4] Tue, 21 Sep 2021 22:23:34 GMT (1274kb,D)
[v5] Thu, 7 Oct 2021 10:25:43 GMT (1273kb,D)

Link back to: arXiv, form interface, contact.