We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation

Abstract: Automatic metrics are commonly used as the exclusive tool for declaring the superiority of one machine translation system's quality over another. The community choice of automatic metric guides research directions and industrial developments by deciding which models are deemed better. Evaluating metrics correlations has been limited to a small collection of human judgements. In this paper, we corroborate how reliable metrics are in contrast to human judgements on - to the best of our knowledge - the largest collection of human judgements. We investigate which metrics have the highest accuracy to make system-level quality rankings for pairs of systems, taking human judgement as a gold standard, which is the closest scenario to the real metric usage. Furthermore, we evaluate the performance of various metrics across different language pairs and domains. Lastly, we show that the sole use of BLEU negatively affected the past development of improved models. We release the collection of human judgements of 4380 systems, and 2.3 M annotated sentences for further analysis and replication of our work.
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2107.10821 [cs.CL]
  (or arXiv:2107.10821v1 [cs.CL] for this version)

Submission history

From: Tom Kocmi [view email]
[v1] Thu, 22 Jul 2021 17:22:22 GMT (2050kb,D)
[v2] Mon, 13 Sep 2021 23:31:16 GMT (2047kb,D)

Link back to: arXiv, form interface, contact.