We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Understanding Learning Dynamics Of Language Models with SVCCA

Abstract: Recent work has demonstrated that neural language models encode linguistic structure implicitly in a number of ways. However, existing research has not shed light on the process by which this structure is acquired during training. We use SVCCA as a tool for understanding how a language model is implicitly predicting a variety of word cluster tags. We present experiments suggesting that a single recurrent layer of a language model learns linguistic structure in phases. We find, for example, that a language model naturally stabilizes its representation of part of speech earlier than it learns semantic and topic information.
Subjects: Computation and Language (cs.CL); Neural and Evolutionary Computing (cs.NE)
Cite as: arXiv:1811.00225 [cs.CL]
  (or arXiv:1811.00225v1 [cs.CL] for this version)

Submission history

From: Naomi Saphra [view email]
[v1] Thu, 1 Nov 2018 04:51:20 GMT (528kb,D)
[v2] Mon, 25 Feb 2019 10:15:06 GMT (1261kb,D)
[v3] Wed, 3 Apr 2019 14:45:07 GMT (1297kb,D)

Link back to: arXiv, form interface, contact.