References & Citations
Computer Science > Computation and Language
Title: MVP: Multi-task Supervised Pre-training for Natural Language Generation
(Submitted on 24 Jun 2022 (this version), latest version 28 May 2023 (v3))
Abstract: Pre-trained language models (PLMs) have achieved notable success in natural language generation (NLG) tasks. Up to now, most of the PLMs are pre-trained in an unsupervised manner using large-scale general corpus. In the meanwhile, an increasing number of models pre-trained with less labeled data showcase superior performance compared to unsupervised models. Motivated by the success of supervised pre-training, we propose Multi-task superVised Pre-training (MVP) for natural language generation. For pre-training the text generation model MVP, we collect a labeled pre-training corpus from 45 datasets over seven generation tasks. For each task, we further pre-train specific soft prompts to stimulate the model capacity in performing a specific task. Extensive experiments have demonstrated the effectiveness of our supervised pre-training in a number of NLG tasks, and our general methods achieve state-of-the-art performance on 12 of 17 datasets.
Submission history
From: Tianyi Tang [view email][v1] Fri, 24 Jun 2022 07:49:47 GMT (235kb,D)
[v2] Mon, 19 Dec 2022 11:44:38 GMT (235kb,D)
[v3] Sun, 28 May 2023 14:41:31 GMT (200kb,D)
Link back to: arXiv, form interface, contact.