References & Citations
Computer Science > Computation and Language
Title: Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation
(Submitted on 25 May 2022 (v1), last revised 23 Oct 2022 (this version, v2))
Abstract: In this paper, we explore the challenging problem of performing a generative task in a target language when labeled data is only available in English, using summarization as a case study. We assume a strict setting with no access to parallel data or machine translation and find that common transfer learning approaches struggle in this setting, as a generative multilingual model fine-tuned purely on English catastrophically forgets how to generate non-English. Given the recent rise of parameter-efficient adaptation techniques, we conduct the first investigation into how one such method, prompt tuning (Lester et al., 2021), can overcome catastrophic forgetting to enable zero-shot cross-lingual generation. Our experiments show that parameter-efficient prompt tuning provides gains over standard fine-tuning when transferring between less-related languages, e.g., from English to Thai. However, a significant gap still remains between these methods and fully-supervised baselines. To improve cross-lingual transfer further, we explore several approaches, including: (1) mixing in unlabeled multilingual data, and (2) explicitly factoring prompts into recombinable language and task components. Our approaches can provide further quality gains, suggesting that robust zero-shot cross-lingual generation is within reach.
Submission history
From: Tu Vu [view email][v1] Wed, 25 May 2022 10:41:34 GMT (7740kb,D)
[v2] Sun, 23 Oct 2022 17:41:30 GMT (996kb,D)
Link back to: arXiv, form interface, contact.