We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Pre-trained Summarization Distillation

Abstract: Current state-of-the-art approaches to summarization utilize large pre-trained Transformer models. Distilling these models to smaller student models has become critically important for practical use; however there are many different distillation methods in the NLP literature. Recent work on distilling BERT for classification and regression tasks shows strong performance using standard knowledge distillation. Alternatively, machine translation practitioners, have primarily distilled using pseudo labeling, where a small model is trained on the translations of a larger model. A third approach is to 'shrink and fine-tune' (SFT), which avoids any explicit distillation by transferring parameters to a student model and then fine-tuning. This work considers distillation of BART and Pegasus, two state of the art summarization models, on two datasets across a variety of student models. We produce high quality, fast checkpoints across different computational budgets, and learn some patterns about which distillation techniques perform well in which situations. PyTorch code to rerun our methods, and use the distilled BART and Pegasus checkpoints is available in Hugging Face transformers.
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as: arXiv:2010.13002 [cs.CL]
  (or arXiv:2010.13002v1 [cs.CL] for this version)

Submission history

From: Sam Shleifer [view email]
[v1] Sat, 24 Oct 2020 23:15:43 GMT (4369kb,D)
[v2] Wed, 28 Oct 2020 04:47:59 GMT (4369kb,D)

Link back to: arXiv, form interface, contact.