Current browse context:
cs.CL
Change to browse by:
References & Citations
Computer Science > Computation and Language
Title: MENLI: Robust Evaluation Metrics from Natural Language Inference
(Submitted on 15 Aug 2022 (v1), last revised 26 Dec 2023 (this version, v5))
Abstract: Recently proposed BERT-based evaluation metrics for text generation perform well on standard benchmarks but are vulnerable to adversarial attacks, e.g., relating to information correctness. We argue that this stems (in part) from the fact that they are models of semantic similarity. In contrast, we develop evaluation metrics based on Natural Language Inference (NLI), which we deem a more appropriate modeling. We design a preference-based adversarial attack framework and show that our NLI based metrics are much more robust to the attacks than the recent BERT-based metrics. On standard benchmarks, our NLI based metrics outperform existing summarization metrics, but perform below SOTA MT metrics. However, when combining existing metrics with our NLI metrics, we obtain both higher adversarial robustness (15%-30%) and higher quality metrics as measured on standard benchmarks (+5% to 30%).
Submission history
From: Yanran Chen [view email][v1] Mon, 15 Aug 2022 16:30:14 GMT (5082kb,D)
[v2] Mon, 3 Apr 2023 16:15:04 GMT (1367kb,D)
[v3] Tue, 4 Apr 2023 10:23:20 GMT (1368kb,D)
[v4] Tue, 11 Apr 2023 15:10:05 GMT (1375kb,D)
[v5] Tue, 26 Dec 2023 04:11:04 GMT (1375kb,D)
Link back to: arXiv, form interface, contact.