We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generator

Abstract: Large-scale pre-trained language models (PLMs) are well-known for being capable of solving a task simply by conditioning a few input-label pairs dubbed demonstrations on a prompt without being explicitly tuned for the desired downstream task. Such a process (i.e., in-context learning), however, naturally leads to high reliance on the demonstrations which are usually selected from external datasets. In this paper, we propose self-generated in-context learning (SG-ICL), which generates demonstrations for in-context learning from PLM itself to minimize the reliance on the external demonstration. We conduct experiments on four different text classification tasks and show SG-ICL significantly outperforms zero-shot learning and is generally worth approximately 0.6 gold training samples. Moreover, our generated demonstrations show more consistent performance with low variance compared to randomly selected demonstrations from the training dataset.
Comments: NAACL 2022 Workshop on Large-scale Pre-trained Language Models
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2206.08082 [cs.CL]
  (or arXiv:2206.08082v1 [cs.CL] for this version)

Submission history

From: Hyuhng Joon Kim [view email]
[v1] Thu, 16 Jun 2022 10:52:13 GMT (3441kb,D)

Link back to: arXiv, form interface, contact.