We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Supervised Graph Contrastive Pretraining for Text Classification

Abstract: Contrastive pretraining techniques for text classification has been largely studied in an unsupervised setting. However, oftentimes labeled data from related tasks which share label semantics with current task is available. We hypothesize that using this labeled data effectively can lead to better generalization on current task. In this paper, we propose a novel way to effectively utilize labeled data from related tasks with a graph based supervised contrastive learning approach. We formulate a token-graph by extrapolating the supervised information from examples to tokens. Our formulation results in an embedding space where tokens with high/low probability of belonging to same class are near/further-away from one another. We also develop detailed theoretical insights which serve as a motivation for our method. In our experiments with $13$ datasets, we show our method outperforms pretraining schemes by $2.5\%$ and also example-level contrastive learning based formulation by $1.8\%$ on average. In addition, we show cross-domain effectiveness of our method in a zero-shot setting by $3.91\%$ on average. Lastly, we also demonstrate our method can be used as a noisy teacher in a knowledge distillation setting to significantly improve performance of transformer based models in low labeled data regime by $4.57\%$ on average.
Comments: A condensed version of this paper has been accepted to ACM SAC'22. DOI: this https URL
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
DOI: 10.1145/3477314.3507194
Cite as: arXiv:2112.11389 [cs.CL]
  (or arXiv:2112.11389v1 [cs.CL] for this version)

Submission history

From: Samujjwal Ghosh [view email]
[v1] Tue, 21 Dec 2021 17:47:14 GMT (122kb,D)

Link back to: arXiv, form interface, contact.