We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Systematic Generalization with Edge Transformers

Abstract: Recent research suggests that systematic generalization in natural language understanding remains a challenge for state-of-the-art neural models such as Transformers and Graph Neural Networks. To tackle this challenge, we propose Edge Transformer, a new model that combines inspiration from Transformers and rule-based symbolic AI. The first key idea in Edge Transformers is to associate vector states with every edge, that is, with every pair of input nodes -- as opposed to just every node, as it is done in the Transformer model. The second major innovation is a triangular attention mechanism that updates edge representations in a way that is inspired by unification from logic programming. We evaluate Edge Transformer on compositional generalization benchmarks in relational reasoning, semantic parsing, and dependency parsing. In all three settings, the Edge Transformer outperforms Relation-aware, Universal and classical Transformer baselines.
Comments: Accepted as a conference paper at NeurIPS 2021
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as: arXiv:2112.00578 [cs.CL]
  (or arXiv:2112.00578v1 [cs.CL] for this version)

Submission history

From: Dzmitry Bahdanau [view email]
[v1] Wed, 1 Dec 2021 15:50:45 GMT (1533kb,D)

Link back to: arXiv, form interface, contact.