We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:


References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: SKDBERT: Compressing BERT via Stochastic Knowledge Distillation

Abstract: In this paper, we propose Stochastic Knowledge Distillation (SKD) to obtain compact BERT-style language model dubbed SKDBERT. In each iteration, SKD samples a teacher model from a pre-defined teacher ensemble, which consists of multiple teacher models with multi-level capacities, to transfer knowledge into student model in an one-to-one manner. Sampling distribution plays an important role in SKD. We heuristically present three types of sampling distributions to assign appropriate probabilities for multi-level teacher models. SKD has two advantages: 1) it can preserve the diversities of multi-level teacher models via stochastically sampling single teacher model in each iteration, and 2) it can also improve the efficacy of knowledge distillation via multi-level teacher models when large capacity gap exists between the teacher model and the student model. Experimental results on GLUE benchmark show that SKDBERT reduces the size of a BERT$_{\rm BASE}$ model by 40% while retaining 99.5% performances of language understanding and being 100% faster.
Comments: This paper has been accepted by AAAI2023
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2211.14466 [cs.CL]
  (or arXiv:2211.14466v2 [cs.CL] for this version)

Submission history

From: Zixiang Ding [view email]
[v1] Sat, 26 Nov 2022 03:18:55 GMT (6073kb,D)
[v2] Tue, 29 Nov 2022 04:12:02 GMT (6059kb,D)

Link back to: arXiv, form interface, contact.