We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus

Abstract: Large text corpora are increasingly important for a wide variety of Natural Language Processing (NLP) tasks, and automatic language identification (LangID) is a core technology needed to collect such datasets in a multilingual context. LangID is largely treated as solved in the literature, with models reported that achieve over 90% average F1 on as many as 1,366 languages. We train LangID models on up to 1,629 languages with comparable quality on held-out test sets, but find that human-judged LangID accuracy for web-crawl text corpora created using these models is only around 5% for many lower-resource languages, suggesting a need for more robust evaluation. Further analysis revealed a variety of error modes, arising from domain mismatch, class imbalance, language similarity, and insufficiently expressive models. We propose two classes of techniques to mitigate these errors: wordlist-based tunable-precision filters (for which we release curated lists in about 500 languages) and transformer-based semi-supervised LangID models, which increase median dataset precision from 5.5% to 71.2%. These techniques enable us to create an initial data set covering 100K or more relatively clean sentences in each of 500+ languages, paving the way towards a 1,000-language web text corpus.
Comments: Accepted to COLING 2020. 9 pages with 8 page abstract
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as: arXiv:2010.14571 [cs.CL]
  (or arXiv:2010.14571v2 [cs.CL] for this version)

Submission history

From: Isaac Caswell [view email]
[v1] Tue, 27 Oct 2020 19:29:17 GMT (1076kb,D)
[v2] Thu, 29 Oct 2020 15:18:35 GMT (1944kb,D)

Link back to: arXiv, form interface, contact.