We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:


References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Learning Bounded Context-Free-Grammar via LSTM and the Transformer:Difference and Explanations

Abstract: Long Short-Term Memory (LSTM) and Transformers are two popular neural architectures used for natural language processing tasks. Theoretical results show that both are Turing-complete and can represent any context-free language (CFL).In practice, it is often observed that Transformer models have better representation power than LSTM. But the reason is barely understood. We study such practical differences between LSTM and Transformer and propose an explanation based on their latent space decomposition patterns. To achieve this goal, we introduce an oracle training paradigm, which forces the decomposition of the latent representation of LSTM and the Transformer and supervises with the transitions of the Pushdown Automaton (PDA) of the corresponding CFL. With the forced decomposition, we show that the performance upper bounds of LSTM and Transformer in learning CFL are close: both of them can simulate a stack and perform stack operation along with state transitions. However, the absence of forced decomposition leads to the failure of LSTM models to capture the stack and stack operations, while having a marginal impact on the Transformer model. Lastly, we connect the experiment on the prototypical PDA to a real-world parsing task to re-verify the conclusions
Comments: Accepted By AAAI22
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2112.09174 [cs.CL]
  (or arXiv:2112.09174v2 [cs.CL] for this version)

Submission history

From: Hui Shi [view email]
[v1] Thu, 16 Dec 2021 19:56:44 GMT (550kb,D)
[v2] Wed, 23 Mar 2022 02:20:19 GMT (558kb,D)

Link back to: arXiv, form interface, contact.