References & Citations
Computer Science > Computation and Language
Title: Near-Negative Distinction: Giving a Second Life to Human Evaluation Datasets
(Submitted on 13 May 2022 (v1), last revised 9 Nov 2022 (this version, v2))
Abstract: Precisely assessing the progress in natural language generation (NLG) tasks is challenging, and human evaluation to establish a preference in a model's output over another is often necessary. However, human evaluation is usually costly, difficult to reproduce, and non-reusable. In this paper, we propose a new and simple automatic evaluation method for NLG called Near-Negative Distinction (NND) that repurposes prior human annotations into NND tests. In an NND test, an NLG model must place a higher likelihood on a high-quality output candidate than on a near-negative candidate with a known error. Model performance is established by the number of NND tests a model passes, as well as the distribution over task-specific errors the model fails on. Through experiments on three NLG tasks (question generation, question answering, and summarization), we show that NND achieves a higher correlation with human judgments than standard NLG evaluation metrics. We then illustrate NND evaluation in four practical scenarios, for example performing fine-grain model analysis, or studying model training dynamics. Our findings suggest that NND can give a second life to human annotations and provide low-cost NLG evaluation.
Submission history
From: Philippe Laban [view email][v1] Fri, 13 May 2022 20:02:53 GMT (519kb,D)
[v2] Wed, 9 Nov 2022 16:48:15 GMT (1040kb,D)
Link back to: arXiv, form interface, contact.