We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: On Learning the Transformer Kernel

Abstract: In this work we introduce KERNELIZED TRANSFORMER, a generic, scalable, data driven framework for learning the kernel function in Transformers. Our framework approximates the Transformer kernel as a dot product between spectral feature maps and learns the kernel by learning the spectral distribution. This not only helps in learning a generic kernel end-to-end, but also reduces the time and space complexity of Transformers from quadratic to linear. We show that KERNELIZED TRANSFORMERS achieve performance comparable to existing efficient Transformer architectures, both in terms of accuracy as well as computational efficiency. Our study also demonstrates that the choice of the kernel has a substantial impact on performance, and kernel learning variants are competitive alternatives to fixed kernel Transformers, both in long as well as short sequence tasks.
Comments: 26 pages, of which 11 form the appendix. 6 figures of which 2 are part of appendix
Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
Cite as: arXiv:2110.08323 [cs.LG]
  (or arXiv:2110.08323v1 [cs.LG] for this version)

Submission history

From: Sankalan Pal Chowdhury [view email]
[v1] Fri, 15 Oct 2021 19:20:25 GMT (9776kb,D)

Link back to: arXiv, form interface, contact.