References & Citations
Computer Science > Computation and Language
Title: An Analysis of GPT-3's Performance in Grammatical Error Correction
(Submitted on 25 Mar 2023 (this version), latest version 30 May 2023 (v2))
Abstract: GPT-3 models are very powerful, achieving high performance on a variety of natural language processing tasks. However, there is a relative lack of detailed published analysis on how well they perform on the task of grammatical error correction (GEC). To address this, we perform experiments testing the capabilities of a GPT-3 model (text-davinci-003) against major GEC benchmarks, comparing the performance of several different prompts, including a comparison of zero-shot and few-shot settings. We analyze intriguing or problematic outputs encountered with different prompt formats. We report the performance of our best prompt on the BEA-2019 and JFLEG datasets using a combination of automatic metrics and human evaluations, revealing interesting differences between the preferences of human raters and the reference-based automatic metrics.
Submission history
From: Steven Coyne [view email][v1] Sat, 25 Mar 2023 03:08:49 GMT (29kb,D)
[v2] Tue, 30 May 2023 12:27:29 GMT (117kb,D)
Link back to: arXiv, form interface, contact.