References & Citations
Computer Science > Computation and Language
Title: $\infty$-former: Infinite Memory Transformer
(Submitted on 1 Sep 2021 (v1), last revised 25 Mar 2022 (this version, v3))
Abstract: Transformers are unable to model long-term memories effectively, since the amount of computation they need to perform grows with the context length. While variations of efficient transformers have been proposed, they all have a finite memory capacity and are forced to drop old information. In this paper, we propose the $\infty$-former, which extends the vanilla transformer with an unbounded long-term memory. By making use of a continuous-space attention mechanism to attend over the long-term memory, the $\infty$-former's attention complexity becomes independent of the context length, trading off memory length with precision. In order to control where precision is more important, $\infty$-former maintains "sticky memories" being able to model arbitrarily long contexts while keeping the computation budget fixed. Experiments on a synthetic sorting task, language modeling, and document grounded dialogue generation demonstrate the $\infty$-former's ability to retain information from long sequences.
Submission history
From: Pedro Henrique Martins [view email][v1] Wed, 1 Sep 2021 10:51:58 GMT (2931kb,D)
[v2] Wed, 15 Sep 2021 10:12:03 GMT (2931kb,D)
[v3] Fri, 25 Mar 2022 10:37:54 GMT (1305kb,D)
Link back to: arXiv, form interface, contact.