We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: How catastrophic can catastrophic forgetting be in linear regression?

Abstract: To better understand catastrophic forgetting, we study fitting an overparameterized linear model to a sequence of tasks with different input distributions. We analyze how much the model forgets the true labels of earlier tasks after training on subsequent tasks, obtaining exact expressions and bounds. We establish connections between continual learning in the linear setting and two other research areas: alternating projections and the Kaczmarz method. In specific settings, we highlight differences between forgetting and convergence to the offline solution as studied in those areas. In particular, when T tasks in d dimensions are presented cyclically for k iterations, we prove an upper bound of T^2 * min{1/sqrt(k), d/k} on the forgetting. This stands in contrast to the convergence to the offline solution, which can be arbitrarily slow according to existing alternating projection results. We further show that the T^2 factor can be lifted when tasks are presented in a random ordering.
Subjects: Machine Learning (cs.LG); Numerical Analysis (math.NA)
Journal reference: 35th Annual Conference on Learning Theory (2022)
Cite as: arXiv:2205.09588 [cs.LG]
  (or arXiv:2205.09588v2 [cs.LG] for this version)

Submission history

From: Itay Evron [view email]
[v1] Thu, 19 May 2022 14:28:40 GMT (440kb,D)
[v2] Wed, 25 May 2022 12:25:57 GMT (440kb,D)

Link back to: arXiv, form interface, contact.