References & Citations
Computer Science > Computation and Language
Title: Universal Sentence Representation Learning with Conditional Masked Language Model
(Submitted on 28 Dec 2020 (v1), last revised 10 Sep 2021 (this version, v3))
Abstract: This paper presents a novel training method, Conditional Masked Language Modeling (CMLM), to effectively learn sentence representations on large scale unlabeled corpora. CMLM integrates sentence representation learning into MLM training by conditioning on the encoded vectors of adjacent sentences. Our English CMLM model achieves state-of-the-art performance on SentEval, even outperforming models learned using supervised signals. As a fully unsupervised learning method, CMLM can be conveniently extended to a broad range of languages and domains. We find that a multilingual CMLM model co-trained with bitext retrieval (BR) and natural language inference (NLI) tasks outperforms the previous state-of-the-art multilingual models by a large margin, e.g. 10% improvement upon baseline models on cross-lingual semantic search. We explore the same language bias of the learned representations, and propose a simple, post-training and model agnostic approach to remove the language identifying information from the representation while still retaining sentence semantics.
Submission history
From: Ziyi Yang [view email][v1] Mon, 28 Dec 2020 18:06:37 GMT (382kb,D)
[v2] Tue, 29 Dec 2020 03:29:11 GMT (382kb,D)
[v3] Fri, 10 Sep 2021 08:18:27 GMT (461kb,D)
Link back to: arXiv, form interface, contact.