We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.AI

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Artificial Intelligence

Title: Making Transformers Solve Compositional Tasks

Abstract: Several studies have reported the inability of Transformer models to generalize compositionally, a key type of generalization in many NLP tasks such as semantic parsing. In this paper we explore the design space of Transformer models showing that the inductive biases given to the model by several design decisions significantly impact compositional generalization. Through this exploration, we identified Transformer configurations that generalize compositionally significantly better than previously reported in the literature in a diverse set of compositional tasks, and that achieve state-of-the-art results in a semantic parsing compositional generalization benchmark (COGS), and a string edit operation composition benchmark (PCFG).
Comments: Source code: this https URL
Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Journal reference: ACL 2022
Cite as: arXiv:2108.04378 [cs.AI]
  (or arXiv:2108.04378v2 [cs.AI] for this version)

Submission history

From: Santiago Ontanon [view email]
[v1] Mon, 9 Aug 2021 22:38:29 GMT (262kb,D)
[v2] Thu, 3 Mar 2022 17:02:15 GMT (267kb,D)

Link back to: arXiv, form interface, contact.