References & Citations
Computer Science > Computation and Language
Title: InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation
(Submitted on 2 Dec 2021 (v1), last revised 25 Mar 2022 (this version, v3))
Abstract: Assessing the quality of natural language generation systems through human annotation is very expensive. Additionally, human annotation campaigns are time-consuming and include non-reusable human labour. In practice, researchers rely on automatic metrics as a proxy of quality. In the last decade, many string-based metrics (e.g., BLEU) have been introduced. However, such metrics usually rely on exact matches and thus, do not robustly handle synonyms. In this paper, we introduce InfoLM a family of untrained metrics that can be viewed as a string-based metric that addresses the aforementioned flaws thanks to a pre-trained masked language model. This family of metrics also makes use of information measures allowing the adaptation of InfoLM to various evaluation criteria. Using direct assessment, we demonstrate that InfoLM achieves statistically significant improvement and over $10$ points of correlation gains in many configurations on both summarization and data2text generation.
Submission history
From: Pierre Colombo [view email][v1] Thu, 2 Dec 2021 20:09:29 GMT (2823kb,D)
[v2] Mon, 13 Dec 2021 16:14:13 GMT (2823kb,D)
[v3] Fri, 25 Mar 2022 14:22:53 GMT (3748kb,D)
Link back to: arXiv, form interface, contact.