We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:


References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: MEGA: Multilingual Evaluation of Generative AI

Abstract: Generative AI models have impressive performance on many Natural Language Processing tasks such as language understanding, reasoning and language generation. One of the most important questions that is being asked by the AI community today is about the capabilities and limits of these models, and it is clear that evaluating generative AI is very challenging. Most studies on generative Large Language Models (LLMs) are restricted to English and it is unclear how capable these models are at understanding and generating other languages. We present the first comprehensive benchmarking of generative LLMs - MEGA, which evaluates models on standard NLP benchmarks, covering 8 diverse tasks and 33 typologically diverse languages. We also compare the performance of generative LLMs to State of the Art (SOTA) non-autoregressive models on these tasks to determine how well generative models perform compared to the previous generation of LLMs. We present a thorough analysis of the performance of models across languages and discuss some of the reasons why generative LLMs are currently not optimal for all languages. We create a framework for evaluating generative LLMs in the multilingual setting and provide directions for future progress in the field.
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2303.12528 [cs.CL]
  (or arXiv:2303.12528v3 [cs.CL] for this version)

Submission history

From: Kabir Ahuja [view email]
[v1] Wed, 22 Mar 2023 13:03:10 GMT (9383kb,D)
[v2] Mon, 3 Apr 2023 05:57:24 GMT (9382kb,D)
[v3] Fri, 19 May 2023 19:20:59 GMT (13588kb,D)

Link back to: arXiv, form interface, contact.