We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Virtual Augmentation Supported Contrastive Learning of Sentence Representations

Abstract: Despite profound successes, contrastive representation learning relies on carefully designed data augmentations using domain specific knowledge. This challenge is magnified in natural language processing where no general rules exist for data augmentation due to the discrete nature of natural language. We tackle this challenge by presenting a Virtual augmentation Supported Contrastive Learning of sentence representations (VaSCL). Originating from the interpretation that data augmentation essentially constructs the neighborhoods of each training instance, we in turn utilize the neighborhood to generate effective data augmentations. Leveraging the large training batch size of contrastive learning, we approximate the neighborhood of an instance via its K-nearest in-batch neighbors in the representation space. We then define an instance discrimination task within this neighborhood, and generate the virtual augmentation in an adversarial training manner. We access the performance of VaSCL on a wide range of downstream tasks, and set a new state-of-the-art for unsupervised sentence representation learning.
Comments: 8 pages, 3 figures, 3 tables
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as: arXiv:2110.08552 [cs.CL]
  (or arXiv:2110.08552v1 [cs.CL] for this version)

Submission history

From: Dejiao Zhang [view email]
[v1] Sat, 16 Oct 2021 11:29:03 GMT (169kb,D)

Link back to: arXiv, form interface, contact.