We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Improving Neural Machine Translation by Denoising Training

Abstract: We present a simple and effective pretraining strategy {D}en{o}ising {T}raining DoT for neural machine translation. Specifically, we update the model parameters with source- and target-side denoising tasks at the early stage and then tune the model normally. Notably, our approach does not increase any parameters or training steps, requiring the parallel data merely. Experiments show that DoT consistently improves the neural machine translation performance across 12 bilingual and 16 multilingual directions (data size ranges from 80K to 20M). In addition, we show that DoT can complement existing data manipulation strategies, i.e. curriculum learning, knowledge distillation, data diversification, bidirectional training, and back-translation. Encouragingly, we found that DoT outperforms costly pretrained model mBART in high-resource settings. Analyses show DoT is a novel in-domain cross-lingual pretraining strategy and could offer further improvements with task-relevant self-supervisions.
Comments: arXiv admin note: text overlap with arXiv:2109.07780
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2201.07365 [cs.CL]
  (or arXiv:2201.07365v2 [cs.CL] for this version)

Submission history

From: Liang Ding [view email]
[v1] Wed, 19 Jan 2022 00:11:38 GMT (50kb,D)
[v2] Thu, 20 Jan 2022 03:55:52 GMT (50kb,D)

Link back to: arXiv, form interface, contact.