We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Encoders and Ensembles for Task-Free Continual Learning

Abstract: We present an architecture that is effective for continual learning in an especially demanding setting, where task boundaries do not exist or are unknown. Our architecture comprises an encoder, pre-trained on a separate dataset, and an ensemble of simple one-layer classifiers. Two main innovations are required to make this combination work. First, the provision of suitably generic pre-trained encoders has been made possible thanks to recent progress in self-supervised training methods. Second, pairing each classifier in the ensemble with a key, where the key-space is identical to the latent space of the encoder, allows them to be used collectively, yet selectively, via k-nearest neighbour lookup. We show that models trained with the encoders-and-ensembles architecture are state-of-the-art for the task-free setting on standard image classification continual learning benchmarks, and improve on prior state-of-the-art by a large margin in the most challenging cases. We also show that the architecture learns well in a fully incremental setting, where one class is learned at a time, and we demonstrate its effectiveness in this setting with up to 100 classes. Finally, we show that the architecture works in a task-free continual learning context where the data distribution changes gradually, and existing approaches requiring knowledge of task boundaries cannot be applied.
Subjects: Machine Learning (cs.LG)
Cite as: arXiv:2105.13327 [cs.LG]
  (or arXiv:2105.13327v2 [cs.LG] for this version)

Submission history

From: Murray Shanahan [view email]
[v1] Thu, 27 May 2021 17:34:31 GMT (2674kb,D)
[v2] Thu, 15 Jul 2021 15:07:13 GMT (1271kb,D)
[v3] Mon, 19 Jul 2021 10:30:31 GMT (1272kb,D)
[v4] Tue, 21 Sep 2021 22:23:34 GMT (1274kb,D)
[v5] Thu, 7 Oct 2021 10:25:43 GMT (1273kb,D)

Link back to: arXiv, form interface, contact.