References & Citations
Computer Science > Computation and Language
Title: A Bilingual Generative Transformer for Semantic Sentence Embedding
(Submitted on 10 Nov 2019 (this version), latest version 19 Nov 2020 (v2))
Abstract: Semantic sentence embedding models encode natural language sentences into vectors, such that closeness in embedding space indicates closeness in the semantics between the sentences. Bilingual data offers a useful signal for learning such embeddings: properties shared by both sentences in a translation pair are likely semantic, while divergent properties are likely stylistic or language-specific. We propose a deep latent variable model that attempts to perform source separation on parallel sentences, isolating what they have in common in a latent semantic vector, and explaining what is left over with language-specific latent vectors. Our proposed approach differs from past work on semantic sentence encoding in two ways. First, by using a variational probabilistic framework, we introduce priors that encourage source separation, and can use our model's posterior to predict sentence embeddings for monolingual data at test time. Second, we use high-capacity transformers as both data generating distributions and inference networks -- contrasting with most past work on sentence embeddings. In experiments, our approach substantially outperforms the state-of-the-art on a standard suite of unsupervised semantic similarity evaluations. Further, we demonstrate that our approach yields the largest gains on more difficult subsets of these evaluations where simple word overlap is not a good indicator of similarity.
Submission history
From: John Wieting [view email][v1] Sun, 10 Nov 2019 10:48:09 GMT (301kb,D)
[v2] Thu, 19 Nov 2020 17:21:10 GMT (305kb,D)
Link back to: arXiv, form interface, contact.