We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: TransDrift: Modeling Word-Embedding Drift using Transformer

Abstract: In modern NLP applications, word embeddings are a crucial backbone that can be readily shared across a number of tasks. However as the text distributions change and word semantics evolve over time, the downstream applications using the embeddings can suffer if the word representations do not conform to the data drift. Thus, maintaining word embeddings to be consistent with the underlying data distribution is a key problem. In this work, we tackle this problem and propose TransDrift, a transformer-based prediction model for word embeddings. Leveraging the flexibility of transformer, our model accurately learns the dynamics of the embedding drift and predicts the future embedding. In experiments, we compare with existing methods and show that our model makes significantly more accurate predictions of the word embedding than the baselines. Crucially, by applying the predicted embeddings as a backbone for downstream classification tasks, we show that our embeddings lead to superior performance compared to the previous methods.
Comments: 10 pages
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as: arXiv:2206.08081 [cs.CL]
  (or arXiv:2206.08081v1 [cs.CL] for this version)

Submission history

From: Nishtha Madaan [view email]
[v1] Thu, 16 Jun 2022 10:48:26 GMT (7256kb,D)

Link back to: arXiv, form interface, contact.