We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Multitask Prompted Training Enables Zero-Shot Task Generalization

Abstract: Large language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks. It has been hypothesized that this is a consequence of implicit multitask learning in language model training. Can zero-shot generalization instead be directly induced by explicit multitask learning? To test this question at scale, we develop a system for easily mapping general natural language tasks into a human-readable prompted form. We convert a large set of supervised datasets, each with multiple prompts using varying natural language. These prompted datasets allow for benchmarking the ability of a model to perform completely unseen tasks specified in natural language. We fine-tune a pretrained encoder-decoder model on this multitask mixture covering a wide variety of tasks. The model attains strong zero-shot performance on several standard datasets, often outperforming models 16x its size. Further, our approach attains strong performance on a subset of tasks from the BIG-Bench benchmark, outperforming models 6x its size. All prompts and trained models are available at github.com/bigscience-workshop/promptsource/.
Comments: this https URL
Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
Cite as: arXiv:2110.08207 [cs.LG]
  (or arXiv:2110.08207v1 [cs.LG] for this version)

Submission history

From: Alexander M. Rush [view email]
[v1] Fri, 15 Oct 2021 17:08:57 GMT (7025kb,D)
[v2] Mon, 13 Dec 2021 03:31:00 GMT (7140kb,D)

Link back to: arXiv, form interface, contact.