We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Contrastive Document Representation Learning with Graph Attention Networks

Abstract: Recent progress in pretrained Transformer-based language models has shown great success in learning contextual representation of text. However, due to the quadratic self-attention complexity, most of the pretrained Transformers models can only handle relatively short text. It is still a challenge when it comes to modeling very long documents. In this work, we propose to use a graph attention network on top of the available pretrained Transformers model to learn document embeddings. This graph attention network allows us to leverage the high-level semantic structure of the document. In addition, based on our graph document model, we design a simple contrastive learning strategy to pretrain our models on a large amount of unlabeled corpus. Empirically, we demonstrate the effectiveness of our approaches in document classification and document retrieval tasks.
Comments: Findings of EMNLP 2021
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2110.10778 [cs.CL]
  (or arXiv:2110.10778v1 [cs.CL] for this version)

Submission history

From: Peng Xu [view email]
[v1] Wed, 20 Oct 2021 21:05:02 GMT (5776kb,D)

Link back to: arXiv, form interface, contact.