We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations

Abstract: Open-domain dialog systems have a user-centric goal: to provide humans with an engaging conversation experience. User engagement is one of the most important metrics for evaluating open-domain dialog systems, and could also be used as real-time feedback to benefit dialog policy learning. Existing work on detecting user disengagement typically requires hand-labeling many dialog samples. We propose HERALD, an efficient annotation framework that reframes the training data annotation process as a denoising problem. Specifically, instead of manually labeling training samples, we first use a set of labeling heuristics to label training samples automatically. We then denoise the weakly labeled data using the Shapley algorithm. Finally, we use the denoised data to train a user engagement detector. Our experiments show that HERALD improves annotation efficiency significantly and achieves 86% user disengagement detection accuracy in two dialog corpora.
Comments: ACL 2021. Code & data available at this https URL
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2106.00162 [cs.CL]
  (or arXiv:2106.00162v2 [cs.CL] for this version)

Submission history

From: Weixin Liang [view email]
[v1] Tue, 1 Jun 2021 01:09:55 GMT (1093kb,D)
[v2] Wed, 2 Jun 2021 06:15:17 GMT (1093kb,D)

Link back to: arXiv, form interface, contact.