References & Citations
Computer Science > Computation and Language
Title: When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute
(Submitted on 24 Feb 2021 (v1), last revised 15 Sep 2021 (this version, v3))
Abstract: Large language models have become increasingly difficult to train because of the growing computation time and cost. In this work, we present SRU++, a highly-efficient architecture that combines fast recurrence and attention for sequence modeling. SRU++ exhibits strong modeling capacity and training efficiency. On standard language modeling tasks such as Enwik8, Wiki-103 and Billion Word datasets, our model obtains better bits-per-character and perplexity while using 3x-10x less training cost compared to top-performing Transformer models. For instance, our model achieves a state-of-the-art result on the Enwik8 dataset using 1.6 days of training on an 8-GPU machine. We further demonstrate that SRU++ requires minimal attention for near state-of-the-art performance. Our results suggest jointly leveraging fast recurrence with little attention as a promising direction for accelerating model training and inference.
Submission history
From: Tao Lei [view email][v1] Wed, 24 Feb 2021 18:39:56 GMT (681kb,D)
[v2] Tue, 30 Mar 2021 16:32:25 GMT (682kb,D)
[v3] Wed, 15 Sep 2021 03:59:10 GMT (939kb,D)
Link back to: arXiv, form interface, contact.