We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: How Does Data Corruption Affect Natural Language Understanding Models? A Study on GLUE datasets

Abstract: A central question in natural language understanding (NLU) research is whether high performance demonstrates the models' strong reasoning capabilities. We present an extensive series of controlled experiments where pre-trained language models are exposed to data that have undergone specific corruption transformations. These involve removing instances of specific word classes and often lead to non-sensical sentences. Our results show that performance remains high on most GLUE tasks when the models are fine-tuned or tested on corrupted data, suggesting that they leverage other cues for prediction even in non-sensical contexts. Our proposed data transformations can be used to assess the extent to which a specific dataset constitutes a proper testbed for evaluating models' language understanding capabilities.
Comments: *SEM 2022 camera ready version
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2201.04467 [cs.CL]
  (or arXiv:2201.04467v2 [cs.CL] for this version)

Submission history

From: Aarne Talman [view email]
[v1] Wed, 12 Jan 2022 13:35:53 GMT (142kb,D)
[v2] Sun, 15 May 2022 10:53:28 GMT (130kb,D)

Link back to: arXiv, form interface, contact.