We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:


References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: A Survey of Pretrained Language Models Based Text Generation

Abstract: Text Generation aims to produce plausible and readable text in human language from input data. The resurgence of deep learning has greatly advanced this field by neural generation models, especially the paradigm of pretrained language models (PLMs). Grounding text generation on PLMs is seen as a promising direction in both academia and industry. In this survey, we present the recent advances achieved in the topic of PLMs for text generation. In detail, we begin with introducing three key points of applying PLMs to text generation: 1) how to encode the input data as representations preserving input semantics which can be fused into PLMs; 2) how to design a universal and performant architecture of PLMs served as generation models; and 3) how to optimize PLMs given the reference text and ensure the generated text satisfying special text properties. Then, we figure out several challenges and future directions within each key point. Next, we present a summary of various useful resources and typical text generation applications to work with PLMs. Finally, we conclude and summarize the contribution of this survey.
Comments: 37 pages, 2 figures, 2 tables
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2201.05273 [cs.CL]
  (or arXiv:2201.05273v2 [cs.CL] for this version)

Submission history

From: Junyi Li [view email]
[v1] Fri, 14 Jan 2022 01:44:58 GMT (160kb,D)
[v2] Wed, 2 Feb 2022 01:34:19 GMT (160kb,D)
[v3] Mon, 28 Feb 2022 14:46:32 GMT (209kb,D)
[v4] Fri, 13 May 2022 18:57:35 GMT (167kb,D)

Link back to: arXiv, form interface, contact.