We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: FLEX: Unifying Evaluation for Few-Shot NLP

Abstract: Few-shot NLP research is highly active, yet conducted in disjoint research threads with evaluation suites that lack challenging-yet-realistic testing setups and fail to employ careful experimental design. Consequently, the community does not know which techniques perform best or even if they outperform simple baselines. In response, we formulate the FLEX Principles, a set of requirements and best practices for unified, rigorous, valid, and cost-sensitive few-shot NLP evaluation. These principles include Sample Size Design, a novel approach to benchmark design that optimizes statistical accuracy and precision while keeping evaluation costs manageable. Following the principles, we release the FLEX benchmark, which includes four few-shot transfer settings, zero-shot evaluation, and a public leaderboard that covers diverse NLP tasks. In addition, we present UniFew, a prompt-based model for few-shot learning that unifies pretraining and finetuning prompt formats, eschewing complex machinery of recent prompt-based approaches in adapting downstream task formats to language model pretraining objectives. We demonstrate that despite simplicity, UniFew achieves results competitive with both popular meta-learning and prompt-based approaches.
Comments: NeurIPS 2021. First two authors contributed equally. Code and leaderboard available at: this https URL
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
ACM classes: I.2.7
Cite as: arXiv:2107.07170 [cs.CL]
  (or arXiv:2107.07170v2 [cs.CL] for this version)

Submission history

From: Jonathan Bragg [view email]
[v1] Thu, 15 Jul 2021 07:37:06 GMT (286kb,D)
[v2] Mon, 8 Nov 2021 18:33:32 GMT (404kb,D)

Link back to: arXiv, form interface, contact.