We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CR

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: As Easy as 1, 2, 3: Behavioural Testing of NMT Systems for Numerical Translation

Abstract: Mistranslated numbers have the potential to cause serious effects, such as financial loss or medical misinformation. In this work we develop comprehensive assessments of the robustness of neural machine translation systems to numerical text via behavioural testing. We explore a variety of numerical translation capabilities a system is expected to exhibit and design effective test examples to expose system underperformance. We find that numerical mistranslation is a general issue: major commercial systems and state-of-the-art research models fail on many of our test examples, for high- and low-resource languages. Our tests reveal novel errors that have not previously been reported in NMT systems, to the best of our knowledge. Lastly, we discuss strategies to mitigate numerical mistranslation.
Comments: Findings of ACL, to appear
Subjects: Computation and Language (cs.CL); Cryptography and Security (cs.CR)
Cite as: arXiv:2107.08357 [cs.CL]
  (or arXiv:2107.08357v1 [cs.CL] for this version)

Submission history

From: Jun Wang [view email]
[v1] Sun, 18 Jul 2021 04:09:47 GMT (203kb,D)

Link back to: arXiv, form interface, contact.