We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CY

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computers and Society

Title: Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs

Abstract: Disaggregated evaluations of AI systems, in which system performance is assessed and reported separately for different groups of people, are conceptually simple. However, their design involves a variety of choices. Some of these choices influence the results that will be obtained, and thus the conclusions that can be drawn; others influence the impacts -- both beneficial and harmful -- that a disaggregated evaluation will have on people, including the people whose data is used to conduct the evaluation. We argue that a deeper understanding of these choices will enable researchers and practitioners to design careful and conclusive disaggregated evaluations. We also argue that better documentation of these choices, along with the underlying considerations and tradeoffs that have been made, will help others when interpreting an evaluation's results and conclusions.
Subjects: Computers and Society (cs.CY); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as: arXiv:2103.06076 [cs.CY]
  (or arXiv:2103.06076v2 [cs.CY] for this version)

Submission history

From: Hanna Wallach [view email]
[v1] Wed, 10 Mar 2021 14:26:14 GMT (67kb,D)
[v2] Wed, 1 Dec 2021 20:38:18 GMT (943kb,D)

Link back to: arXiv, form interface, contact.