We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:


References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants

Abstract: In Dynamic Adversarial Data Collection (DADC), human annotators are tasked with finding examples that models struggle to predict correctly. Models trained on DADC-collected training data have been shown to be more robust in adversarial and out-of-domain settings, and are considerably harder for humans to fool. However, DADC is more time-consuming than traditional data collection and thus more costly per example. In this work, we examine if we can maintain the advantages of DADC, without suffering the additional cost. To that end, we introduce Generative Annotation Assistants (GAAs), generator-in-the-loop models that provide real-time suggestions that annotators can either approve, modify, or reject entirely. We collect training datasets in twenty experimental settings and perform a detailed analysis of this approach for the task of extractive question answering (QA) for both standard and adversarial data collection. We demonstrate that GAAs provide significant efficiency benefits in terms of annotation speed, while leading to improved model fooling rates. In addition, we show that GAA-assisted data leads to higher downstream model performance on a variety of question answering tasks.
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2112.09062 [cs.CL]
  (or arXiv:2112.09062v1 [cs.CL] for this version)

Submission history

From: Max Bartolo [view email]
[v1] Thu, 16 Dec 2021 17:59:39 GMT (292kb,D)
[v2] Tue, 10 May 2022 18:08:10 GMT (587kb,D)
[v3] Tue, 17 May 2022 11:26:43 GMT (597kb,D)

Link back to: arXiv, form interface, contact.