References & Citations
Computer Science > Computation and Language
Title: Adaptive Transformers for Learning Multimodal Representations
(Submitted on 15 May 2020 (v1), last revised 8 Jul 2020 (this version, v3))
Abstract: The usage of transformers has grown from learning about language semantics to forming meaningful visiolinguistic representations. These architectures are often over-parametrized, requiring large amounts of computation. In this work, we extend adaptive approaches to learn more about model interpretability and computational efficiency. Specifically, we study attention spans, sparse, and structured dropout methods to help understand how their attention mechanism extends for vision and language tasks. We further show that these approaches can help us learn more about how the network perceives the complexity of input sequences, sparsity preferences for different modalities, and other related phenomena.
Submission history
From: Prajjwal Bhargava [view email][v1] Fri, 15 May 2020 12:12:57 GMT (635kb,D)
[v2] Wed, 10 Jun 2020 06:40:52 GMT (1304kb,D)
[v3] Wed, 8 Jul 2020 12:26:12 GMT (1302kb,D)
Link back to: arXiv, form interface, contact.