We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Autoencoding Improves Pre-trained Word Embeddings

Abstract: Prior work investigating the geometry of pre-trained word embeddings have shown that word embeddings to be distributed in a narrow cone and by centering and projecting using principal component vectors one can increase the accuracy of a given set of pre-trained word embeddings. However, theoretically, this post-processing step is equivalent to applying a linear autoencoder to minimise the squared l2 reconstruction error. This result contradicts prior work (Mu and Viswanath, 2018) that proposed to remove the top principal components from pre-trained embeddings. We experimentally verify our theoretical claims and show that retaining the top principal components is indeed useful for improving pre-trained word embeddings, without requiring access to additional linguistic resources or labelled data.
Comments: COLING 2020
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2010.13094 [cs.CL]
  (or arXiv:2010.13094v2 [cs.CL] for this version)

Submission history

From: Masahiro Kaneko [view email]
[v1] Sun, 25 Oct 2020 11:30:05 GMT (753kb,D)
[v2] Tue, 27 Oct 2020 07:51:34 GMT (351kb,D)

Link back to: arXiv, form interface, contact.