References & Citations
Computer Science > Computation and Language
Title: Discovering Language-neutral Sub-networks in Multilingual Language Models
(Submitted on 25 May 2022 (v1), last revised 30 Oct 2022 (this version, v2))
Abstract: Multilingual pre-trained language models transfer remarkably well on cross-lingual downstream tasks. However, the extent to which they learn language-neutral representations (i.e., shared representations that encode similar phenomena across languages), and the effect of such representations on cross-lingual transfer performance, remain open questions. In this work, we conceptualize language neutrality of multilingual models as a function of the overlap between language-encoding sub-networks of these models. We employ the lottery ticket hypothesis to discover sub-networks that are individually optimized for various languages and tasks. Our evaluation across three distinct tasks and eleven typologically-diverse languages demonstrates that sub-networks for different languages are topologically similar (i.e., language-neutral), making them effective initializations for cross-lingual transfer with limited performance degradation.
Submission history
From: Negar Foroutan [view email][v1] Wed, 25 May 2022 11:35:41 GMT (1442kb,D)
[v2] Sun, 30 Oct 2022 18:46:56 GMT (5957kb,D)
Link back to: arXiv, form interface, contact.