References & Citations
Computer Science > Computation and Language
Title: SemGloVe: Semantic Co-occurrences for GloVe from BERT
(Submitted on 30 Dec 2020 (v1), last revised 24 Nov 2021 (this version, v2))
Abstract: GloVe learns word embeddings by leveraging statistical information from word co-occurrence matrices. However, word pairs in the matrices are extracted from a predefined local context window, which might lead to limited word pairs and potentially semantic irrelevant word pairs. In this paper, we propose SemGloVe, which distills semantic co-occurrences from BERT into static GloVe word embeddings. Particularly, we propose two models to extract co-occurrence statistics based on either the masked language model or the multi-head attention weights of BERT. Our methods can extract word pairs without limiting by the local window assumption and can define the co-occurrence weights by directly considering the semantic distance between word pairs. Experiments on several word similarity datasets and four external tasks show that SemGloVe can outperform GloVe.
Submission history
From: Leilei Gan [view email][v1] Wed, 30 Dec 2020 15:38:26 GMT (1103kb,D)
[v2] Wed, 24 Nov 2021 07:50:57 GMT (1097kb,D)
Link back to: arXiv, form interface, contact.