References & Citations
Computer Science > Computation and Language
Title: Measuring the Measuring Tools: An Automatic Evaluation of Semantic Metrics for Text Corpora
(Submitted on 29 Nov 2022)
Abstract: The ability to compare the semantic similarity between text corpora is important in a variety of natural language processing applications. However, standard methods for evaluating these metrics have yet to be established. We propose a set of automatic and interpretable measures for assessing the characteristics of corpus-level semantic similarity metrics, allowing sensible comparison of their behavior. We demonstrate the effectiveness of our evaluation measures in capturing fundamental characteristics by evaluating them on a collection of classical and state-of-the-art metrics. Our measures revealed that recently-developed metrics are becoming better in identifying semantic distributional mismatch while classical metrics are more sensitive to perturbations in the surface text levels.
Submission history
From: Samuel Ackerman [view email][v1] Tue, 29 Nov 2022 14:47:07 GMT (17885kb,D)
Link back to: arXiv, form interface, contact.