We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.AI

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Human-Computer Interaction

Title: A Test for Evaluating Performance in Human-Computer Systems

Abstract: The Turing test for comparing computer performance to that of humans is well known, but, surprisingly, there is no widely used test for comparing how much better human-computer systems perform relative to humans alone, computers alone, or other baselines. Here, we show how to perform such a test using the ratio of means as a measure of effect size. Then we demonstrate the use of this test in three ways. First, in an analysis of 79 recently published experimental results, we find that, surprisingly, over half of the studies find a decrease in performance, the mean and median ratios of performance improvement are both approximately 1 (corresponding to no improvement at all), and the maximum ratio is 1.36 (a 36% improvement). Second, we experimentally investigate whether a higher performance improvement ratio is obtained when 100 human programmers generate software using GPT-3, a massive, state-of-the-art AI system. In this case, we find a speed improvement ratio of 1.27 (a 27% improvement). Finally, we find that 50 human non-programmers using GPT-3 can perform the task about as well as--and less expensively than--the human programmers. In this case, neither the non-programmers nor the computer would have been able to perform the task alone, so this is an example of a very strong form of human-computer synergy.
Comments: Corrected typos and references
Subjects: Human-Computer Interaction (cs.HC); Artificial Intelligence (cs.AI); Computers and Society (cs.CY)
ACM classes: C.4; D.2.2; D.2.8; H.1.2; H.5.2; H.5.3; I.2.2
Report number: MIT Center for Collective Intelligence Working Paper No. 2022-001
Cite as: arXiv:2206.12390 [cs.HC]
  (or arXiv:2206.12390v2 [cs.HC] for this version)

Submission history

From: Thomas Malone [view email]
[v1] Fri, 24 Jun 2022 17:44:58 GMT (678kb,D)
[v2] Tue, 28 Jun 2022 19:46:29 GMT (761kb,D)

Link back to: arXiv, form interface, contact.