References & Citations
Computer Science > Computation and Language
Title: Blockwise Self-Attention for Long Document Understanding
(Submitted on 7 Nov 2019 (v1), last revised 1 Nov 2020 (this version, v2))
Abstract: We present BlockBERT, a lightweight and efficient BERT model for better modeling long-distance dependencies. Our model extends BERT by introducing sparse block structures into the attention matrix to reduce both memory consumption and training/inference time, which also enables attention heads to capture either short- or long-range contextual information. We conduct experiments on language model pre-training and several benchmark question answering datasets with various paragraph lengths. BlockBERT uses 18.7-36.1% less memory and 12.0-25.1% less time to learn the model. During testing, BlockBERT saves 27.8% inference time, while having comparable and sometimes better prediction accuracy, compared to an advanced BERT-based model, RoBERTa.
Submission history
From: Jiezhong Qiu [view email][v1] Thu, 7 Nov 2019 16:35:53 GMT (1053kb,D)
[v2] Sun, 1 Nov 2020 12:48:03 GMT (972kb,D)
Link back to: arXiv, form interface, contact.