References & Citations
Computer Science > Computation and Language
Title: Autoencoding Improves Pre-trained Word Embeddings
(Submitted on 25 Oct 2020 (v1), last revised 27 Oct 2020 (this version, v2))
Abstract: Prior work investigating the geometry of pre-trained word embeddings have shown that word embeddings to be distributed in a narrow cone and by centering and projecting using principal component vectors one can increase the accuracy of a given set of pre-trained word embeddings. However, theoretically, this post-processing step is equivalent to applying a linear autoencoder to minimise the squared l2 reconstruction error. This result contradicts prior work (Mu and Viswanath, 2018) that proposed to remove the top principal components from pre-trained embeddings. We experimentally verify our theoretical claims and show that retaining the top principal components is indeed useful for improving pre-trained word embeddings, without requiring access to additional linguistic resources or labelled data.
Submission history
From: Masahiro Kaneko [view email][v1] Sun, 25 Oct 2020 11:30:05 GMT (753kb,D)
[v2] Tue, 27 Oct 2020 07:51:34 GMT (351kb,D)
Link back to: arXiv, form interface, contact.