References & Citations
Computer Science > Computation and Language
Title: Faithful Chain-of-Thought Reasoning
(Submitted on 31 Jan 2023 (v1), last revised 20 Sep 2023 (this version, v3))
Abstract: While Chain-of-Thought (CoT) prompting boosts Language Models' (LM) performance on a gamut of complex reasoning tasks, the generated reasoning chain does not necessarily reflect how the model arrives at the answer (aka. faithfulness). We propose Faithful CoT, a reasoning framework involving two stages: Translation (Natural Language query $\rightarrow$ symbolic reasoning chain) and Problem Solving (reasoning chain $\rightarrow$ answer), using an LM and a deterministic solver respectively. This guarantees that the reasoning chain provides a faithful explanation of the final answer. Aside from interpretability, Faithful CoT also improves empirical performance: it outperforms standard CoT on 9 of 10 benchmarks from 4 diverse domains, with a relative accuracy gain of 6.3% on Math Word Problems (MWP), 3.4% on Planning, 5.5% on Multi-hop Question Answering (QA), and 21.4% on Relational Inference. Furthermore, with GPT-4 and Codex, it sets the new state-of-the-art few-shot performance on 7 datasets (with 95.0+ accuracy on 6 of them), showing a strong synergy between faithfulness and accuracy.
Submission history
From: Qing Lyu [view email][v1] Tue, 31 Jan 2023 03:04:26 GMT (966kb,D)
[v2] Wed, 1 Feb 2023 19:25:24 GMT (966kb,D)
[v3] Wed, 20 Sep 2023 22:19:30 GMT (1405kb,D)
Link back to: arXiv, form interface, contact.