We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization

Abstract: Factual consistency is an essential quality of text summarization models in practical settings. Existing work in evaluating this dimension can be broadly categorized into two lines of research, entailment-based and question answering (QA)-based metrics, and different experimental setups often lead to contrasting conclusions as to which paradigm performs the best. In this work, we conduct an extensive comparison of entailment and QA-based metrics, demonstrating that carefully choosing the components of a QA-based metric, especially question generation and answerability classification, is critical to performance. Building on those insights, we propose an optimized metric, which we call QAFactEval, that leads to a 14% average improvement over previous QA-based metrics on the SummaC factual consistency benchmark, and also outperforms the best-performing entailment-based metric. Moreover, we find that QA-based and entailment-based metrics can offer complementary signals and be combined into a single metric for a further performance boost.
Comments: NAACL 2022
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2112.08542 [cs.CL]
  (or arXiv:2112.08542v2 [cs.CL] for this version)

Submission history

From: Alexander Fabbri [view email]
[v1] Thu, 16 Dec 2021 00:38:35 GMT (6202kb,D)
[v2] Fri, 29 Apr 2022 16:02:14 GMT (6216kb,D)

Link back to: arXiv, form interface, contact.