Current browse context:
cs.CL
Change to browse by:
References & Citations
Computer Science > Computation and Language
Title: CERT: Contrastive Self-supervised Learning for Language Understanding
(Submitted on 16 May 2020 (v1), last revised 18 Jun 2020 (this version, v2))
Abstract: Pretrained language models such as BERT, GPT have shown great effectiveness in language understanding. The auxiliary predictive tasks in existing pretraining approaches are mostly defined on tokens, thus may not be able to capture sentence-level semantics very well. To address this issue, we propose CERT: Contrastive self-supervised Encoder Representations from Transformers, which pretrains language representation models using contrastive self-supervised learning at the sentence level. CERT creates augmentations of original sentences using back-translation. Then it finetunes a pretrained language encoder (e.g., BERT) by predicting whether two augmented sentences originate from the same sentence. CERT is simple to use and can be flexibly plugged into any pretraining-finetuning NLP pipeline. We evaluate CERT on 11 natural language understanding tasks in the GLUE benchmark where CERT outperforms BERT on 7 tasks, achieves the same performance as BERT on 2 tasks, and performs worse than BERT on 2 tasks. On the averaged score of the 11 tasks, CERT outperforms BERT. The data and code are available at this https URL
Submission history
From: Hongchao Fang [view email][v1] Sat, 16 May 2020 16:20:38 GMT (233kb,D)
[v2] Thu, 18 Jun 2020 12:47:18 GMT (289kb,D)
Link back to: arXiv, form interface, contact.