We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses

Abstract: How related are the representations learned by neural language models, translation models, and language tagging tasks? We answer this question by adapting an encoder-decoder transfer learning method from computer vision to investigate the structure among 100 different feature spaces extracted from hidden representations of various networks trained on language tasks. This method reveals a low-dimensional structure where language models and translation models smoothly interpolate between word embeddings, syntactic and semantic tasks, and future word embeddings. We call this low-dimensional structure a language representation embedding because it encodes the relationships between representations needed to process language for a variety of NLP tasks. We find that this representation embedding can predict how well each individual feature space maps to human brain responses to natural language stimuli recorded using fMRI. Additionally, we find that the principal dimension of this structure can be used to create a metric which highlights the brain's natural language processing hierarchy. This suggests that the embedding captures some part of the brain's natural language representation structure.
Comments: Preprint, submitted for review
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as: arXiv:2106.05426 [cs.CL]
  (or arXiv:2106.05426v3 [cs.CL] for this version)

Submission history

From: Richard Antonello [view email]
[v1] Wed, 9 Jun 2021 22:59:12 GMT (12545kb,D)
[v2] Tue, 15 Jun 2021 19:31:26 GMT (12545kb,D)
[v3] Tue, 20 Jul 2021 21:43:24 GMT (12545kb,D)

Link back to: arXiv, form interface, contact.