Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Compressing Large-Scale Transformer-Based Models: A Case Study on BERT
(Submitted on 27 Feb 2020 (v1), last revised 2 Jun 2021 (this version, v2))
Abstract: Pre-trained Transformer-based models have achieved state-of-the-art performance for various Natural Language Processing (NLP) tasks. However, these models often have billions of parameters, and, thus, are too resource-hungry and computation-intensive to suit low-capability devices or applications with strict latency requirements. One potential remedy for this is model compression, which has attracted a lot of research attention. Here, we summarize the research in compressing Transformers, focusing on the especially popular BERT model. In particular, we survey the state of the art in compression for BERT, we clarify the current best practices for compressing large-scale Transformer models, and we provide insights into the workings of various methods. Our categorization and analysis also shed light on promising future research directions for achieving lightweight, accurate, and generic NLP models.
Submission history
From: Prakhar Ganesh [view email][v1] Thu, 27 Feb 2020 09:20:31 GMT (994kb,D)
[v2] Wed, 2 Jun 2021 02:38:20 GMT (1433kb,D)
Link back to: arXiv, form interface, contact.