We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Cross-Task Generalization via Natural Language Crowdsourcing Instructions

Abstract: Humans (e.g., crowdworkers) have a remarkable ability in solving different tasks, by simply reading textual instructions that define them and looking at a few examples. NLP models built with the conventional paradigm, however, often struggle with generalization across tasks (e.g., a question-answering system cannot solve classification tasks). A long-standing challenge in AI is to build a model that learns a new task by understanding the human-readable instructions that define it. To study this, we introduce NATURAL INSTRUCTIONS, a dataset of 61 distinct tasks, their human-authored instructions and 193k task instances. The instructions are obtained from crowdsourcing instructions used to create existing NLP datasets and mapped to a unified schema. We adopt generative pre-trained language models to encode task-specific instructions along with input and generate task output. Our results indicate that models benefit from instructions when evaluated in terms of generalization to unseen tasks. These models, however, are far behind supervised task-specific models, indicating significant room for more progress in this direction.
Comments: 20 pages
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
Cite as: arXiv:2104.08773 [cs.CL]
  (or arXiv:2104.08773v3 [cs.CL] for this version)

Submission history

From: Swaroop Mishra [view email]
[v1] Sun, 18 Apr 2021 08:44:56 GMT (8437kb,D)
[v2] Fri, 3 Sep 2021 21:58:23 GMT (12093kb,D)
[v3] Sat, 16 Oct 2021 05:12:48 GMT (12212kb,D)
[v4] Mon, 14 Mar 2022 09:15:08 GMT (12506kb,D)

Link back to: arXiv, form interface, contact.