We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:


References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: $\infty$-former: Infinite Memory Transformer

Abstract: Transformers struggle when attending to long contexts, since the amount of computation grows with the context length, and therefore they cannot model long-term memories effectively. Several variations have been proposed to alleviate this problem, but they all have a finite memory capacity, being forced to drop old information. In this paper, we propose the $\infty$-former, which extends the vanilla transformer with an unbounded long-term memory. By making use of a continuous-space attention mechanism to attend over the long-term memory, the $\infty$-former's attention complexity becomes independent of the context length. Thus, it is able to model arbitrarily long contexts and maintain "sticky memories" while keeping a fixed computation budget. Experiments on a synthetic sorting task demonstrate the ability of the $\infty$-former to retain information from long sequences. We also perform experiments on language modeling, by training a model from scratch and by fine-tuning a pre-trained language model, which show benefits of unbounded long-term memories.
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2109.00301 [cs.CL]
  (or arXiv:2109.00301v1 [cs.CL] for this version)

Submission history

From: Pedro Henrique Martins [view email]
[v1] Wed, 1 Sep 2021 10:51:58 GMT (2931kb,D)
[v2] Wed, 15 Sep 2021 10:12:03 GMT (2931kb,D)
[v3] Fri, 25 Mar 2022 10:37:54 GMT (1305kb,D)

Link back to: arXiv, form interface, contact.