We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo

Computer Science > Computation and Language

Title: Reproducibility Issues for BERT-based Evaluation Metrics

Abstract: Reproducibility is of utmost concern in machine learning and natural language processing (NLP). In the field of natural language generation (especially machine translation), the seminal paper of Post (2018) has pointed out problems of reproducibility of the dominant metric, BLEU, at the time of publication. Nowadays, BERT-based evaluation metrics considerably outperform BLEU. In this paper, we ask whether results and claims from four recent BERT-based metrics can be reproduced. We find that reproduction of claims and results often fails because of (i) heavy undocumented preprocessing involved in the metrics, (ii) missing code and (iii) reporting weaker results for the baseline metrics. (iv) In one case, the problem stems from correlating not to human scores but to a wrong column in the csv file, inflating scores by 5 points. Motivated by the impact of preprocessing, we then conduct a second study where we examine its effects more closely (for one of the metrics). We find that preprocessing can have large effects, especially for highly inflectional languages. In this case, the effect of preprocessing may be larger than the effect of the aggregation mechanism (e.g., greedy alignment vs. Word Mover Distance).
Comments: EMNLP 2022 Camera-Ready (captions fixed)
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as: arXiv:2204.00004 [cs.CL]
  (or arXiv:2204.00004v3 [cs.CL] for this version)

Submission history

From: Yanran Chen [view email]
[v1] Wed, 30 Mar 2022 20:35:37 GMT (3809kb,D)
[v2] Fri, 21 Oct 2022 18:49:45 GMT (7930kb,D)
[v3] Wed, 26 Oct 2022 20:33:46 GMT (8578kb,D)

Link back to: arXiv, form interface, contact.