References & Citations
Computer Science > Computation and Language
Title: MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models
(Submitted on 9 Nov 2019 (v1), last revised 30 Apr 2020 (this version, v2))
Abstract: Pretrained language models have led to significant performance gains in many NLP tasks. However, the intensive computing resources to train such models remain an issue. Knowledge distillation alleviates this problem by learning a light-weight student model. So far the distillation approaches are all task-specific. In this paper, we explore knowledge distillation under the multi-task learning setting. The student is jointly distilled across different tasks. It acquires more general representation capacity through multi-tasking distillation and can be further fine-tuned to improve the model in the target domain. Unlike other BERT distillation methods which specifically designed for Transformer-based architectures, we provide a general learning framework. Our approach is model agnostic and can be easily applied on different future teacher model architectures. We evaluate our approach on a Transformer-based and LSTM based student model. Compared to a strong, similarly LSTM-based approach, we achieve better quality under the same computational constraints. Compared to the present state of the art, we reach comparable results with much faster inference speed.
Submission history
From: Linqing Liu [view email][v1] Sat, 9 Nov 2019 00:22:05 GMT (105kb,D)
[v2] Thu, 30 Apr 2020 23:19:01 GMT (118kb,D)
Link back to: arXiv, form interface, contact.