References & Citations
Computer Science > Computation and Language
Title: Composable Sparse Fine-Tuning for Cross-Lingual Transfer
(Submitted on 14 Oct 2021 (v1), last revised 9 Feb 2023 (this version, v2))
Abstract: Fine-tuning the entire set of parameters of a large pretrained model has become the mainstream approach for transfer learning. To increase its efficiency and prevent catastrophic forgetting and interference, techniques like adapters and sparse fine-tuning have been developed. Adapters are modular, as they can be combined to adapt a model towards different facets of knowledge (e.g., dedicated language and/or task adapters). Sparse fine-tuning is expressive, as it controls the behavior of all model components. In this work, we introduce a new fine-tuning method with both these desirable properties. In particular, we learn sparse, real-valued masks based on a simple variant of the Lottery Ticket Hypothesis. Task-specific masks are obtained from annotated data in a source language, and language-specific masks from masked language modeling in a target language. Both these masks can then be composed with the pretrained model. Unlike adapter-based fine-tuning, this method neither increases the number of parameters at inference time nor alters the original model architecture. Most importantly, it outperforms adapters in zero-shot cross-lingual transfer by a large margin in a series of multilingual benchmarks, including Universal Dependencies, MasakhaNER, and AmericasNLI. Based on an in-depth analysis, we additionally find that sparsity is crucial to prevent both 1) interference between the fine-tunings to be composed and 2) overfitting. We release the code and models at this https URL
Submission history
From: Alan Ansell [view email][v1] Thu, 14 Oct 2021 17:27:29 GMT (8447kb,D)
[v2] Thu, 9 Feb 2023 10:54:10 GMT (8460kb,D)
Link back to: arXiv, form interface, contact.