References & Citations
Computer Science > Computation and Language
Title: CoRe: An Efficient Coarse-refined Training Framework for BERT
(Submitted on 27 Nov 2020 (v1), last revised 18 Feb 2021 (this version, v2))
Abstract: In recent years, BERT has made significant breakthroughs on many natural language processing tasks and attracted great attentions. Despite its accuracy gains, the BERT model generally involves a huge number of parameters and needs to be trained on massive datasets, so training such a model is computationally very challenging and time-consuming. Hence, training efficiency should be a critical issue. In this paper, we propose a novel coarse-refined training framework named CoRe to speed up the training of BERT. Specifically, we decompose the training process of BERT into two phases. In the first phase, by introducing fast attention mechanism and decomposing the large parameters in the feed-forward network sub-layer, we construct a relaxed BERT model which has much less parameters and much lower model complexity than the original BERT, so the relaxed model can be quickly trained. In the second phase, we transform the trained relaxed BERT model into the original BERT and further retrain the model. Thanks to the desired initialization provided by the relaxed model, the retraining phase requires much less training steps, compared with training an original BERT model from scratch with a random initialization. Experimental results show that the proposed CoRe framework can greatly reduce the training time without reducing the performance.
Submission history
From: Shengnan Wang [view email][v1] Fri, 27 Nov 2020 09:49:37 GMT (1521kb,D)
[v2] Thu, 18 Feb 2021 03:39:00 GMT (1519kb,D)
Link back to: arXiv, form interface, contact.