Current browse context:
cs.CL
Change to browse by:
References & Citations
Computer Science > Computation and Language
Title: Reasoning Like Program Executors
(Submitted on 27 Jan 2022 (v1), last revised 22 Oct 2022 (this version, v2))
Abstract: Reasoning over natural language is a long-standing goal for the research community. However, studies have shown that existing language models are inadequate in reasoning. To address the issue, we present POET, a novel reasoning pre-training paradigm. Through pre-training language models with programs and their execution results, POET empowers language models to harvest the reasoning knowledge possessed by program executors via a data-driven approach. POET is conceptually simple and can be instantiated by different kinds of program executors. In this paper, we showcase two simple instances POET-Math and POET-Logic, in addition to a complex instance, POET-SQL. Experimental results on six benchmarks demonstrate that POET can significantly boost model performance in natural language reasoning, such as numerical reasoning, logical reasoning, and multi-hop reasoning. POET opens a new gate on reasoning-enhancement pre-training, and we hope our analysis would shed light on the future research of reasoning like program executors.
Submission history
From: Qian Liu [view email][v1] Thu, 27 Jan 2022 12:28:24 GMT (5482kb,D)
[v2] Sat, 22 Oct 2022 13:46:24 GMT (5689kb,D)
Link back to: arXiv, form interface, contact.