We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: AI and the Everything in the Whole Wide World Benchmark

Abstract: There is a tendency across different subfields in AI to valorize a small collection of influential benchmarks. These benchmarks operate as stand-ins for a range of anointed common problems that are frequently framed as foundational milestones on the path towards flexible and generalizable AI systems. State-of-the-art performance on these benchmarks is widely understood as indicative of progress towards these long-term goals. In this position paper, we explore the limits of such benchmarks in order to reveal the construct validity issues in their framing as the functionally "general" broad measures of progress they are set up to be.
Comments: Accepted in NeurIPS 2021 Benchmarks and Datasets track
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Performance (cs.PF)
Cite as: arXiv:2111.15366 [cs.LG]
  (or arXiv:2111.15366v1 [cs.LG] for this version)

Submission history

From: Inioluwa Deborah Raji [view email]
[v1] Fri, 26 Nov 2021 19:39:34 GMT (8856kb,D)

Link back to: arXiv, form interface, contact.