We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:


References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: A Survey of Pretrained Language Models Based Text Generation

Abstract: Text Generation aims to produce plausible and readable text in human language from input data. The resurgence of deep learning has greatly advanced this field by neural generation models, especially the paradigm of pretrained language models (PLMs). Text generation based on PLMs is viewed as a promising area in both academics and industry. In this survey, we begin with introducing three key aspects of applying PLMs to text generation: 1) how to encode the input as representations preserving input semantics which can be fused into PLMs; 2) how to design an effective and performant PLM served as the generation model; and 3) how to effectively optimize PLMs given the reference text and ensure the generated text satisfying special text properties. Then, we figure out some major challenges and solutions corresponding to the three key views. Next, we present a summary of various useful resources and typical text generation applications to work with PLMs. Finally, we highlight some of the future research directions which will further improve these PLMs for text generation. We strongly believe that this comprehensive survey paper will serve as a valuable resource to learn the core concepts as well as stay up to date on the latest developments in PLMs.
Comments: 37 pages, 2 figures, 2 tables
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2201.05273 [cs.CL]
  (or arXiv:2201.05273v3 [cs.CL] for this version)

Submission history

From: Junyi Li [view email]
[v1] Fri, 14 Jan 2022 01:44:58 GMT (160kb,D)
[v2] Wed, 2 Feb 2022 01:34:19 GMT (160kb,D)
[v3] Mon, 28 Feb 2022 14:46:32 GMT (209kb,D)
[v4] Fri, 13 May 2022 18:57:35 GMT (167kb,D)

Link back to: arXiv, form interface, contact.