References & Citations
Computer Science > Computation and Language
Title: Helpfulness and Fairness of Task-Oriented Dialogue Systems
(Submitted on 25 May 2022 (v1), last revised 7 May 2023 (this version, v3))
Abstract: Goal-oriented dialogue systems aim to help users achieve certain goals. Therefore, how humans perceive their helpfulness is important. However, neither the human-perceived helpfulness of goal-oriented dialogue systems nor its fairness implication has been well studied. In this paper, we study computational measurements of helpfulness. We first formally define a dialogue response as helpful if it is relevant & coherent, useful, and informative to a query. Then, we collect human annotations for the helpfulness of dialogue responses based on our definition and build a classifier to automatically determine the helpfulness of a response. We further propose to use the helpfulness level of a dialogue system towards different user queries to measure the fairness of a dialogue system. Experiments with state-of-the-art dialogue systems under three information-seeking scenarios reveal that existing systems tend to be more helpful for questions regarding concepts from highly-developed countries than less-developed countries, uncovering potential fairness concerns underlying the current goal-oriented dialogue systems.
Submission history
From: Yu Hou [view email][v1] Wed, 25 May 2022 07:58:38 GMT (7277kb,D)
[v2] Thu, 13 Oct 2022 08:30:05 GMT (7036kb,D)
[v3] Sun, 7 May 2023 08:39:00 GMT (0kb,I)
Link back to: arXiv, form interface, contact.