We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Probing for Incremental Parse States in Autoregressive Language Models

Abstract: Next-word predictions from autoregressive neural language models show remarkable sensitivity to syntax. This work evaluates the extent to which this behavior arises as a result of a learned ability to maintain implicit representations of incremental syntactic structures. We extend work in syntactic probing to the incremental setting and present several probes for extracting incomplete syntactic structure (operationalized through parse states from a stack-based parser) from autoregressive language models. We find that our probes can be used to predict model preferences on ambiguous sentence prefixes and causally intervene on model representations and steer model behavior. This suggests implicit incremental syntactic inferences underlie next-word predictions in autoregressive neural language models.
Comments: Findings of EMNLP 2022
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as: arXiv:2211.09748 [cs.CL]
  (or arXiv:2211.09748v1 [cs.CL] for this version)

Submission history

From: Tiwalayo Eisape [view email]
[v1] Thu, 17 Nov 2022 18:15:31 GMT (3857kb,D)

Link back to: arXiv, form interface, contact.