We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Which transformer architecture fits my data? A vocabulary bottleneck in self-attention

Abstract: After their successful debut in natural language processing, Transformer architectures are now becoming the de-facto standard in many domains. An obstacle for their deployment over new modalities is the architectural configuration: the optimal depth-to-width ratio has been shown to dramatically vary across data types (e.g., $10$x larger over images than over language). We theoretically predict the existence of an embedding rank bottleneck that limits the contribution of self-attention width to the Transformer expressivity. We thus directly tie the input vocabulary size and rank to the optimal depth-to-width ratio, since a small vocabulary size or rank dictates an added advantage of depth over width. We empirically demonstrate the existence of this bottleneck and its implications on the depth-to-width interplay of Transformer architectures, linking the architecture variability across domains to the often glossed-over usage of different vocabulary sizes or embedding ranks in different domains. As an additional benefit, our rank bottlenecking framework allows us to identify size redundancies of $25\%-50\%$ in leading NLP models such as ALBERT and T5.
Comments: ICML 2021
Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
Cite as: arXiv:2105.03928 [cs.LG]
  (or arXiv:2105.03928v2 [cs.LG] for this version)

Submission history

From: Noam Wies [view email]
[v1] Sun, 9 May 2021 13:08:26 GMT (1092kb,D)
[v2] Wed, 9 Jun 2021 17:18:03 GMT (997kb,D)

Link back to: arXiv, form interface, contact.