We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Soft Layer Selection with Meta-Learning for Zero-Shot Cross-Lingual Transfer

Abstract: Multilingual pre-trained contextual embedding models (Devlin et al., 2019) have achieved impressive performance on zero-shot cross-lingual transfer tasks. Finding the most effective fine-tuning strategy to fine-tune these models on high-resource languages so that it transfers well to the zero-shot languages is a non-trivial task. In this paper, we propose a novel meta-optimizer to soft-select which layers of the pre-trained model to freeze during fine-tuning. We train the meta-optimizer by simulating the zero-shot transfer scenario. Results on cross-lingual natural language inference show that our approach improves over the simple fine-tuning baseline and X-MAML (Nooralahzadeh et al., 2020).
Comments: MetaNLP at ACL 2021
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2107.09840 [cs.CL]
  (or arXiv:2107.09840v1 [cs.CL] for this version)

Submission history

From: Weijia Xu [view email]
[v1] Wed, 21 Jul 2021 02:16:45 GMT (121kb,D)

Link back to: arXiv, form interface, contact.