We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Natural Instructions: Benchmarking Generalization to New Tasks from Natural Language Instructions

Abstract: Can we enable NLP models to appropriately respond to instructional prompts and consequently generalize to new tasks? To study this question, we leverage the existing NLP datasets and the instructions that were used to crowdsource them to create NATURAL INSTRUCTIONS, a dataset of instructions and task-specific input/output data. This dataset consists of 61 distinct language instructions and about 600k task instances, and is used to evaluate existing state-of-the-art language-models (LMs) in addressing new tasks by few-shot prompting of GPT3 and fine-tuning BART. Our analysis indicates that: (a) the existing models indeed benefit from instructions and hence, show improved generalization to new tasks; (b) while models like GPT-3 generally benefit from instructions, the extent of their gains varies across different fields of instructions and also depends on the task being solved; (c) generalization to unseen tasks in NATURAL INSTRUCTIONS remains far from perfect for the state-of-the-art, indicating significant room for more progress in this direction.
Comments: 18 pages
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
Cite as: arXiv:2104.08773 [cs.CL]
  (or arXiv:2104.08773v1 [cs.CL] for this version)

Submission history

From: Swaroop Mishra [view email]
[v1] Sun, 18 Apr 2021 08:44:56 GMT (8437kb,D)
[v2] Fri, 3 Sep 2021 21:58:23 GMT (12093kb,D)
[v3] Sat, 16 Oct 2021 05:12:48 GMT (12212kb,D)
[v4] Mon, 14 Mar 2022 09:15:08 GMT (12506kb,D)

Link back to: arXiv, form interface, contact.