We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Of Non-Linearity and Commutativity in BERT

Abstract: In this work we provide new insights into the transformer architecture, and in particular, its best-known variant, BERT. First, we propose a method to measure the degree of non-linearity of different elements of transformers. Next, we focus our investigation on the feed-forward networks (FFN) inside transformers, which contain 2/3 of the model parameters and have so far not received much attention. We find that FFNs are an inefficient yet important architectural element and that they cannot simply be replaced by attention blocks without a degradation in performance. Moreover, we study the interactions between layers in BERT and show that, while the layers exhibit some hierarchical structure, they extract features in a fuzzy manner. Our results suggest that BERT has an inductive bias towards layer commutativity, which we find is mainly due to the skip connections. This provides a justification for the strong performance of recurrent and weight-shared transformer models.
Comments: Accepted at IJCNN 2021
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as: arXiv:2101.04547 [cs.CL]
  (or arXiv:2101.04547v4 [cs.CL] for this version)

Submission history

From: Damian Pascual [view email]
[v1] Tue, 12 Jan 2021 15:29:38 GMT (608kb,D)
[v2] Wed, 13 Jan 2021 09:58:42 GMT (608kb,D)
[v3] Thu, 14 Jan 2021 10:23:01 GMT (608kb,D)
[v4] Fri, 7 May 2021 07:38:46 GMT (1342kb,D)

Link back to: arXiv, form interface, contact.