We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:


References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models

Abstract: In this paper, we detail the relationship between convolutions and self-attention in natural language tasks. We show that relative position embeddings in self-attention layers are equivalent to recently-proposed dynamic lightweight convolutions, and we consider multiple new ways of integrating convolutions into Transformer self-attention. Specifically, we propose composite attention, which unites previous relative position embedding methods under a convolutional framework. We conduct experiments by training BERT with composite attention, finding that convolutions consistently improve performance on multiple downstream tasks, replacing absolute position embeddings. To inform future work, we present results comparing lightweight convolutions, dynamic convolutions, and depthwise-separable convolutions in language model pre-training, considering multiple injection points for convolutions in self-attention layers.
Comments: Accepted to ACL-IJCNLP 2021
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2106.05505 [cs.CL]
  (or arXiv:2106.05505v1 [cs.CL] for this version)

Submission history

From: Tyler A. Chang [view email]
[v1] Thu, 10 Jun 2021 05:11:35 GMT (120kb,D)

Link back to: arXiv, form interface, contact.