We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Word Frequency Does Not Predict Grammatical Knowledge in Language Models

Abstract: Neural language models learn, to varying degrees of accuracy, the grammatical properties of natural languages. In this work, we investigate whether there are systematic sources of variation in the language models' accuracy. Focusing on subject-verb agreement and reflexive anaphora, we find that certain nouns are systematically understood better than others, an effect which is robust across grammatical tasks and different language models. Surprisingly, we find that across four orders of magnitude, corpus frequency is unrelated to a noun's performance on grammatical tasks. Finally, we find that a novel noun's grammatical properties can be few-shot learned from various types of training data. The results present a paradox: there should be less variation in grammatical performance than is actually observed.
Comments: EMNLP 2020
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2010.13870 [cs.CL]
  (or arXiv:2010.13870v1 [cs.CL] for this version)

Submission history

From: Leon Bergen [view email]
[v1] Mon, 26 Oct 2020 19:51:36 GMT (8298kb,D)

Link back to: arXiv, form interface, contact.