Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Knowledge-Adaptation Priors
(Submitted on 16 Jun 2021 (v1), last revised 27 Oct 2021 (this version, v2))
Abstract: Humans and animals have a natural ability to quickly adapt to their surroundings, but machine-learning models, when subjected to changes, often require a complete retraining from scratch. We present Knowledge-adaptation priors (K-priors) to reduce the cost of retraining by enabling quick and accurate adaptation for a wide-variety of tasks and models. This is made possible by a combination of weight and function-space priors to reconstruct the gradients of the past, which recovers and generalizes many existing, but seemingly-unrelated, adaptation strategies. Training with simple first-order gradient methods can often recover the exact retrained model to an arbitrary accuracy by choosing a sufficiently large memory of the past data. Empirical results show that adaptation with K-priors achieves performance similar to full retraining, but only requires training on a handful of past examples.
Submission history
From: Siddharth Swaroop [view email][v1] Wed, 16 Jun 2021 13:27:22 GMT (2621kb,D)
[v2] Wed, 27 Oct 2021 13:46:22 GMT (1216kb,D)
Link back to: arXiv, form interface, contact.