Current browse context:
cs.CL
Change to browse by:
References & Citations
Computer Science > Computation and Language
Title: GEMv2: Multilingual NLG Benchmarking in a Single Line of Code
(Submitted on 22 Jun 2022 (v1), last revised 24 Jun 2022 (this version, v3))
Abstract: Evaluation in machine learning is usually informed by past choices, for example which datasets or metrics to use. This standardization enables the comparison on equal footing using leaderboards, but the evaluation choices become sub-optimal as better alternatives arise. This problem is especially pertinent in natural language generation which requires ever-improving suites of datasets, metrics, and human evaluation to make definitive claims. To make following best model evaluation practices easier, we introduce GEMv2. The new version of the Generation, Evaluation, and Metrics Benchmark introduces a modular infrastructure for dataset, model, and metric developers to benefit from each others work. GEMv2 supports 40 documented datasets in 51 languages. Models for all datasets can be evaluated online and our interactive data card creation and rendering tools make it easier to add new datasets to the living benchmark.
Submission history
From: Sebastian Gehrmann [view email][v1] Wed, 22 Jun 2022 17:52:30 GMT (7696kb,D)
[v2] Thu, 23 Jun 2022 14:38:38 GMT (7696kb,D)
[v3] Fri, 24 Jun 2022 12:48:24 GMT (7696kb,D)
Link back to: arXiv, form interface, contact.