References & Citations
Computer Science > Machine Learning
Title: Attention Approximates Sparse Distributed Memory
(Submitted on 10 Nov 2021 (v1), last revised 17 Jan 2022 (this version, v2))
Abstract: While Attention has come to be an important mechanism in deep learning, there remains limited intuition for why it works so well. Here, we show that Transformer Attention can be closely related under certain data conditions to Kanerva's Sparse Distributed Memory (SDM), a biologically plausible associative memory model. We confirm that these conditions are satisfied in pre-trained GPT2 Transformer models. We discuss the implications of the Attention-SDM map and provide new computational and biological interpretations of Attention.
Submission history
From: Trenton Bricken [view email][v1] Wed, 10 Nov 2021 02:36:32 GMT (14769kb,D)
[v2] Mon, 17 Jan 2022 18:00:00 GMT (14769kb,D)
Link back to: arXiv, form interface, contact.