We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Understanding Learning Dynamics Of Language Models with SVCCA

Abstract: Research has shown that neural models implicitly encode linguistic features, but there has been no research showing \emph{how} these encodings arise as the models are trained. We present the first study on the learning dynamics of neural language models, using a simple and flexible analysis method called Singular Vector Canonical Correlation Analysis (SVCCA), which enables us to compare learned representations across time and across models, without the need to evaluate directly on annotated data. We probe the evolution of syntactic, semantic, and topic representations and find that part-of-speech is learned earlier than topic; that recurrent layers become more similar to those of a tagger during training; and embedding layers less similar. Our results and methods could inform better learning algorithms for NLP models, possibly to incorporate linguistic information more effectively.
Comments: Accepted for publication in NAACL 2019
Subjects: Computation and Language (cs.CL); Neural and Evolutionary Computing (cs.NE)
DOI: 10.18653/v1/N19-1329
Cite as: arXiv:1811.00225 [cs.CL]
  (or arXiv:1811.00225v3 [cs.CL] for this version)

Submission history

From: Naomi Saphra [view email]
[v1] Thu, 1 Nov 2018 04:51:20 GMT (528kb,D)
[v2] Mon, 25 Feb 2019 10:15:06 GMT (1261kb,D)
[v3] Wed, 3 Apr 2019 14:45:07 GMT (1297kb,D)

Link back to: arXiv, form interface, contact.