References & Citations
Computer Science > Computation and Language
Title: Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translation
(Submitted on 10 Aug 2022 (v1), last revised 5 Mar 2023 (this version, v2))
Abstract: Although the problem of hallucinations in neural machine translation (NMT) has received some attention, research on this highly pathological phenomenon lacks solid ground. Previous work has been limited in several ways: it often resorts to artificial settings where the problem is amplified, it disregards some (common) types of hallucinations, and it does not validate adequacy of detection heuristics. In this paper, we set foundations for the study of NMT hallucinations. First, we work in a natural setting, i.e., in-domain data without artificial noise neither in training nor in inference. Next, we annotate a dataset of over 3.4k sentences indicating different kinds of critical errors and hallucinations. Then, we turn to detection methods and both revisit methods used previously and propose using glass-box uncertainty-based detectors. Overall, we show that for preventive settings, (i) previously used methods are largely inadequate, (ii) sequence log-probability works best and performs on par with reference-based methods. Finally, we propose DeHallucinator, a simple method for alleviating hallucinations at test time that significantly reduces the hallucinatory rate. To ease future research, we release our annotated dataset for WMT18 German-English data, along with the model, training data, and code.
Submission history
From: Nuno Miguel Guerreiro [view email][v1] Wed, 10 Aug 2022 12:44:13 GMT (5648kb,D)
[v2] Sun, 5 Mar 2023 09:28:23 GMT (5654kb,D)
Link back to: arXiv, form interface, contact.