References & Citations
Computer Science > Computation and Language
Title: Code Prompting: a Neural Symbolic Method for Complex Reasoning in Large Language Models
(Submitted on 29 May 2023 (v1), last revised 7 Oct 2023 (this version, v2))
Abstract: Large language models (LLMs) have scaled up to unlock a wide range of complex reasoning tasks with the aid of various prompting methods. However, current prompting methods generate natural language intermediate steps to help reasoning, which can cause imperfect task reduction and confusion. To mitigate such limitations, we explore code prompting, a neural symbolic prompting method with both zero-shot and few-shot versions which triggers code as intermediate steps. We conduct experiments on 7 widely-used benchmarks involving symbolic reasoning and arithmetic reasoning. Code prompting generally outperforms chain-of-thought (CoT) prompting. To further understand the performance and limitations of code prompting, we perform extensive ablation studies and error analyses, and identify several exclusive advantages of using symbolic promptings compared to natural language. We also consider the ensemble of code prompting and CoT prompting to combine the strengths of both. Finally, we show through experiments how code annotations and their locations affect code prompting.
Submission history
From: Yi Hu [view email][v1] Mon, 29 May 2023 15:14:09 GMT (1901kb,D)
[v2] Sat, 7 Oct 2023 08:07:46 GMT (2175kb,D)
Link back to: arXiv, form interface, contact.