References & Citations
Computer Science > Computation and Language
Title: BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretraining
(Submitted on 31 Dec 2020 (v1), last revised 16 Jun 2021 (this version, v3))
Abstract: In this paper, we propose BANG, a new pretraining model to Bridge the gap between Autoregressive (AR) and Non-autoregressive (NAR) Generation. AR and NAR generation can be uniformly regarded as to what extent previous tokens can be attended, and BANG bridges AR and NAR generation by designing a novel model structure for large-scale pretraining. The pretrained BANG model can simultaneously support AR, NAR and semi-NAR generation to meet different requirements. Experiments on question generation (SQuAD 1.1), summarization (XSum) and dialogue generation (PersonaChat) show that BANG improves NAR and semi-NAR performance significantly as well as attaining comparable performance with strong AR pretrained models. Compared with the semi-NAR strong baselines, BANG achieves absolute improvements of 14.01 and 5.24 in the overall scores of SQuAD 1.1 and XSum, respectively. In addition, BANG achieves absolute improvements of 10.73, 6.39 and 5.90 in the overall scores of SQuAD, XSUM and PersonaChat respectively compared with the strong NAR baselines.
Submission history
From: Weizhen Qi [view email][v1] Thu, 31 Dec 2020 10:09:29 GMT (968kb,D)
[v2] Thu, 18 Feb 2021 11:12:31 GMT (259kb,D)
[v3] Wed, 16 Jun 2021 06:11:08 GMT (1089kb,D)
Link back to: arXiv, form interface, contact.