References & Citations
Computer Science > Computation and Language
Title: A Primer on Pretrained Multilingual Language Models
(Submitted on 1 Jul 2021 (v1), last revised 23 Dec 2021 (this version, v2))
Abstract: Multilingual Language Models (\MLLMs) such as mBERT, XLM, XLM-R, \textit{etc.} have emerged as a viable option for bringing the power of pretraining to a large number of languages. Given their success in zero-shot transfer learning, there has emerged a large body of work in (i) building bigger \MLLMs~covering a large number of languages (ii) creating exhaustive benchmarks covering a wider variety of tasks and languages for evaluating \MLLMs~ (iii) analysing the performance of \MLLMs~on monolingual, zero-shot cross-lingual and bilingual tasks (iv) understanding the universal language patterns (if any) learnt by \MLLMs~ and (v) augmenting the (often) limited capacity of \MLLMs~ to improve their performance on seen or even unseen languages. In this survey, we review the existing literature covering the above broad areas of research pertaining to \MLLMs. Based on our survey, we recommend some promising directions of future research.
Submission history
From: Sumanth Doddapaneni [view email][v1] Thu, 1 Jul 2021 18:01:46 GMT (99kb,D)
[v2] Thu, 23 Dec 2021 09:51:27 GMT (360kb,D)
Link back to: arXiv, form interface, contact.