References & Citations
Computer Science > Computation and Language
Title: Hey AI, Can You Solve Complex Tasks by Talking to Agents?
(Submitted on 16 Oct 2021 (v1), last revised 9 May 2022 (this version, v2))
Abstract: Training giant models from scratch for each complex task is resource- and data-inefficient. To help develop models that can leverage existing systems, we propose a new challenge: Learning to solve complex tasks by communicating with existing agents (or models) in natural language. We design a synthetic benchmark, CommaQA, with three complex reasoning tasks (explicit, implicit, numeric) designed to be solved by communicating with existing QA agents. For instance, using text and table QA agents to answer questions such as "Who had the longest javelin throw from USA?". We show that black-box models struggle to learn this task from scratch (accuracy under 50\%) even with access to each agent's knowledge and gold facts supervision. In contrast, models that learn to communicate with agents outperform black-box models, reaching scores of 100\% when given gold decomposition supervision. However, we show that the challenge of learning to solve complex tasks by communicating with existing agents \emph{without relying on any auxiliary supervision or data} still remains highly elusive. We release CommaQA, along with a compositional generalization test split, to advance research in this direction. Dataset and Code available at this https URL
Submission history
From: Tushar Khot [view email][v1] Sat, 16 Oct 2021 10:37:34 GMT (392kb,D)
[v2] Mon, 9 May 2022 18:15:36 GMT (879kb,D)
Link back to: arXiv, form interface, contact.