We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.HC

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Human-Computer Interaction

Title: CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities

Abstract: Large language models (LMs) offer unprecedented language generation capabilities and exciting opportunities for interaction design. However, their highly context-dependent capabilities are difficult to grasp and are often subjectively interpreted. In this paper, we argue that by curating and analyzing large interaction datasets, the HCI community can foster more incisive examinations of LMs' generative capabilities. Exemplifying this approach, we present CoAuthor, a dataset designed for revealing GPT-3's capabilities in assisting creative and argumentative writing. CoAuthor captures rich interactions between 63 writers and four instances of GPT-3 across 1445 writing sessions. We demonstrate that CoAuthor can address questions about GPT-3's language, ideation, and collaboration capabilities, and reveal its contribution as a writing "collaborator" under various definitions of good collaboration. Finally, we discuss how this work may facilitate a more principled discussion around LMs' promises and pitfalls in relation to interaction design. The dataset and an interface for replaying the writing sessions are publicly available at this https URL
Comments: Published as a conference paper at CHI 2022
Subjects: Human-Computer Interaction (cs.HC); Computation and Language (cs.CL)
DOI: 10.1145/3491102.3502030
Cite as: arXiv:2201.06796 [cs.HC]
  (or arXiv:2201.06796v2 [cs.HC] for this version)

Submission history

From: Mina Lee [view email]
[v1] Tue, 18 Jan 2022 07:51:57 GMT (1906kb,D)
[v2] Tue, 25 Jan 2022 05:29:58 GMT (1907kb,D)

Link back to: arXiv, form interface, contact.