References & Citations
Computer Science > Computation and Language
Title: Sparsifying Transformer Models with Trainable Representation Pooling
(Submitted on 10 Sep 2020 (v1), revised 4 Feb 2021 (this version, v3), latest version 7 Mar 2022 (v4))
Abstract: We propose a novel method to sparsify attention in the Transformer model by learning to select the most-informative token representations during the training process, thus focusing on task-specific parts of the input. A reduction of quadratic time and memory complexity to sublinear was achieved due to a robust trainable top-k operator. For example, our experiments on a challenging summarization task of long documents show that our method is over 3 times faster and up to 16 times more memory efficient while significantly outperforming both dense and state-of-the-art sparse transformer models. The method can be effortlessly applied to many models used in NLP and CV, simultaneously with other improvements.
Submission history
From: Lukasz Borchmann [view email][v1] Thu, 10 Sep 2020 22:49:39 GMT (158kb,D)
[v2] Thu, 14 Jan 2021 16:49:01 GMT (263kb,D)
[v3] Thu, 4 Feb 2021 22:34:47 GMT (975kb,D)
[v4] Mon, 7 Mar 2022 12:49:08 GMT (1010kb,D)
Link back to: arXiv, form interface, contact.