We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Context-Tuning: Learning Contextualized Prompts for Natural Language Generation

Abstract: Recently, pretrained language models (PLMs) have made exceptional success in language generation. To leverage the rich knowledge encoded by PLMs, a simple yet powerful mechanism is to use prompts, in the form of either discrete tokens or continuous embeddings. In existing studies, manual prompts are time-consuming and require domain expertise, while continuous prompts are typically independent of the inputs. To address this issue, we propose a novel continuous prompting approach, called Context-Tuning, to fine-tuning PLMs for natural language generation. Firstly, the prompts are derived based on the input text, so that they can elicit useful knowledge from PLMs for generation. We refer to such prompts as contextualized prompts. Secondly, to further enhance the relevance of the generated text to the inputs, we utilize continuous inverse prompting to refine the process of natural language generation by modeling an inverse generation process from output to input. Moreover, we propose a lightweight contexttuning, fine-tuning only 0.4% of parameters while retaining well performance.
Comments: 13 pages, 6 figures, 6 tables
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2201.08670 [cs.CL]
  (or arXiv:2201.08670v1 [cs.CL] for this version)

Submission history

From: Tianyi Tang [view email]
[v1] Fri, 21 Jan 2022 12:35:28 GMT (168kb,D)

Link back to: arXiv, form interface, contact.