References & Citations
Computer Science > Computation and Language
Title: Undivided Attention: Are Intermediate Layers Necessary for BERT?
(Submitted on 22 Dec 2020 (v1), last revised 4 Apr 2023 (this version, v2))
Abstract: In recent times, BERT-based models have been extremely successful in solving a variety of natural language processing (NLP) tasks such as reading comprehension, natural language inference, sentiment analysis, etc. All BERT-based architectures have a self-attention block followed by a block of intermediate layers as the basic building component. However, a strong justification for the inclusion of these intermediate layers remains missing in the literature. In this work we investigate the importance of intermediate layers on the overall network performance of downstream tasks. We show that reducing the number of intermediate layers and modifying the architecture for BERT-BASE results in minimal loss in fine-tuning accuracy for downstream tasks while decreasing the number of parameters and training time of the model. Additionally, we use centered kernel alignment and probing linear classifiers to gain insight into our architectural modifications and justify that removal of intermediate layers has little impact on the fine-tuned accuracy.
Submission history
From: Anthony Sarah [view email][v1] Tue, 22 Dec 2020 08:46:14 GMT (149kb,D)
[v2] Tue, 4 Apr 2023 21:10:29 GMT (87kb,D)
Link back to: arXiv, form interface, contact.