References & Citations
Computer Science > Computation and Language
Title: QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
(Submitted on 16 Dec 2021 (v1), last revised 29 Apr 2022 (this version, v2))
Abstract: Factual consistency is an essential quality of text summarization models in practical settings. Existing work in evaluating this dimension can be broadly categorized into two lines of research, entailment-based and question answering (QA)-based metrics, and different experimental setups often lead to contrasting conclusions as to which paradigm performs the best. In this work, we conduct an extensive comparison of entailment and QA-based metrics, demonstrating that carefully choosing the components of a QA-based metric, especially question generation and answerability classification, is critical to performance. Building on those insights, we propose an optimized metric, which we call QAFactEval, that leads to a 14% average improvement over previous QA-based metrics on the SummaC factual consistency benchmark, and also outperforms the best-performing entailment-based metric. Moreover, we find that QA-based and entailment-based metrics can offer complementary signals and be combined into a single metric for a further performance boost.
Submission history
From: Alexander Fabbri [view email][v1] Thu, 16 Dec 2021 00:38:35 GMT (6202kb,D)
[v2] Fri, 29 Apr 2022 16:02:14 GMT (6216kb,D)
Link back to: arXiv, form interface, contact.