We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Small-Bench NLP: Benchmark for small single GPU trained models in Natural Language Processing

Abstract: Recent progress in the Natural Language Processing domain has given us several State-of-the-Art (SOTA) pretrained models which can be finetuned for specific tasks. These large models with billions of parameters trained on numerous GPUs/TPUs over weeks are leading in the benchmark leaderboards. In this paper, we discuss the need for a benchmark for cost and time effective smaller models trained on a single GPU. This will enable researchers with resource constraints experiment with novel and innovative ideas on tokenization, pretraining tasks, architecture, fine tuning methods etc. We set up Small-Bench NLP, a benchmark for small efficient neural language models trained on a single GPU. Small-Bench NLP benchmark comprises of eight NLP tasks on the publicly available GLUE datasets and a leaderboard to track the progress of the community. Our ELECTRA-DeBERTa (15M parameters) small model architecture achieves an average score of 81.53 which is comparable to that of BERT-Base's 82.20 (110M parameters). Our models, code and leaderboard are available at this https URL
Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
Cite as: arXiv:2109.10847 [cs.LG]
  (or arXiv:2109.10847v2 [cs.LG] for this version)

Submission history

From: Kamal Raj Kanakarajan [view email]
[v1] Wed, 22 Sep 2021 17:18:55 GMT (52kb,D)
[v2] Thu, 23 Sep 2021 06:19:05 GMT (52kb,D)

Link back to: arXiv, form interface, contact.