We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:


References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: The Rediscovery Hypothesis: Language Models Need to Meet Linguistics

Abstract: There is an ongoing debate in the NLP community whether modern language models contain linguistic knowledge, recovered through so-called probes. In this paper, we study whether linguistic knowledge is a necessary condition for the good performance of modern language models, which we call the \textit{rediscovery hypothesis}. In the first place, we show that language models that are significantly compressed but perform well on their pretraining objectives retain good scores when probed for linguistic structures. This result supports the rediscovery hypothesis and leads to the second contribution of our paper: an information-theoretic framework that relates language modeling objectives with linguistic information. This framework also provides a metric to measure the impact of linguistic information on the word prediction task. We reinforce our analytical results with various experiments, both on synthetic and on real NLP tasks in English.
Subjects: Computation and Language (cs.CL)
Journal reference: Journal of Artificial Intelligence Vol. 72 (2021) 1343-1384
DOI: 10.1613/jair.1.12788
Cite as: arXiv:2103.01819 [cs.CL]
  (or arXiv:2103.01819v2 [cs.CL] for this version)

Submission history

From: Matthias Gallé [view email]
[v1] Tue, 2 Mar 2021 15:57:39 GMT (4611kb,D)
[v2] Mon, 3 Jan 2022 07:31:01 GMT (6020kb,D)

Link back to: arXiv, form interface, contact.