References & Citations
Computer Science > Computation and Language
Title: A Transformer with Stack Attention
(Submitted on 7 May 2024 (v1), last revised 13 May 2024 (this version, v2))
Abstract: Natural languages are believed to be (mildly) context-sensitive. Despite underpinning remarkably capable large language models, transformers are unable to model many context-free language tasks. In an attempt to address this limitation in the modeling power of transformer-based language models, we propose augmenting them with a differentiable, stack-based attention mechanism. Our stack-based attention mechanism can be incorporated into any transformer-based language model and adds a level of interpretability to the model. We show that the addition of our stack-based attention mechanism enables the transformer to model some, but not all, deterministic context-free languages.
Submission history
From: Jiaoda Li [view email][v1] Tue, 7 May 2024 17:47:57 GMT (304kb,D)
[v2] Mon, 13 May 2024 18:56:18 GMT (304kb,D)
Link back to: arXiv, form interface, contact.