We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: A Contrastive Framework for Neural Text Generation

Abstract: Text generation is of great importance to many natural language processing applications. However, maximization-based decoding methods (e.g. beam search) of neural language models often lead to degenerate solutions -- the generated text is unnatural and contains undesirable repetitions. Existing approaches introduce stochasticity via sampling or modify training objectives to decrease probabilities of certain tokens (e.g., unlikelihood training). However, they often lead to solutions that lack coherence. In this work, we show that an underlying reason for model degeneration is the anisotropic distribution of token representations. We present a contrastive solution: (i) SimCTG, a contrastive training objective to calibrate the model's representation space, and (ii) a decoding method -- contrastive search -- to encourage diversity while maintaining coherence in the generated text. Extensive experiments and analyses on three benchmarks from two languages demonstrate that our proposed approach outperforms state-of-the-art text generation methods as evaluated by both human and automatic metrics.
Comments: 22 pages, 8 figures, and 10 tables (v2 adds some experimental results)
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2202.06417 [cs.CL]
  (or arXiv:2202.06417v2 [cs.CL] for this version)

Submission history

From: Yixuan Su [view email]
[v1] Sun, 13 Feb 2022 21:46:14 GMT (5592kb,D)
[v2] Mon, 30 May 2022 08:25:58 GMT (2809kb,D)
[v3] Mon, 26 Sep 2022 14:59:44 GMT (2854kb,D)

Link back to: arXiv, form interface, contact.