We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research

Abstract: Transparency around limitations can improve the scientific rigor of research, help ensure appropriate interpretation of research findings, and make research claims more credible. Despite these benefits, the machine learning (ML) research community lacks well-developed norms around disclosing and discussing limitations. To address this gap, we conduct an iterative design process with 30 ML and ML-adjacent researchers to develop and test REAL ML, a set of guided activities to help ML researchers recognize, explore, and articulate the limitations of their research. Using a three-stage interview and survey study, we identify ML researchers' perceptions of limitations, as well as the challenges they face when recognizing, exploring, and articulating limitations. We develop REAL ML to address some of these practical challenges, and highlight additional cultural challenges that will require broader shifts in community norms to address. We hope our study and REAL ML help move the ML research community toward more active and appropriate engagement with limitations.
Comments: This work appears in the 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT '22)
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computers and Society (cs.CY)
DOI: 10.1145/3531146.3533122
Cite as: arXiv:2205.08363 [cs.LG]
  (or arXiv:2205.08363v1 [cs.LG] for this version)

Submission history

From: Jessie J. Smith [view email]
[v1] Thu, 5 May 2022 15:32:45 GMT (8731kb,D)

Link back to: arXiv, form interface, contact.