We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:


References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Evaluation of contextual embeddings on less-resourced languages

Abstract: The current dominance of deep neural networks in natural language processing is based on contextual embeddings such as ELMo, BERT, and BERT derivatives. Most existing work focuses on English; in contrast, we present here the first multilingual empirical comparison of two ELMo and several monolingual and multilingual BERT models using 14 tasks in nine languages. In monolingual settings, our analysis shows that monolingual BERT models generally dominate, with a few exceptions such as the dependency parsing task, where they are not competitive with ELMo models trained on large corpora. In cross-lingual settings, BERT models trained on only a few languages mostly do best, closely followed by massively multilingual BERT models.
Comments: 45 pages
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2107.10614 [cs.CL]
  (or arXiv:2107.10614v1 [cs.CL] for this version)

Submission history

From: Matej Ulčar [view email]
[v1] Thu, 22 Jul 2021 12:32:27 GMT (107kb)

Link back to: arXiv, form interface, contact.