References & Citations
Computer Science > Computation and Language
Title: What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks
(Submitted on 27 May 2023 (v1), last revised 28 Dec 2023 (this version, v3))
Abstract: Large Language Models (LLMs) with strong abilities in natural language processing tasks have emerged and have been applied in various kinds of areas such as science, finance and software engineering. However, the capability of LLMs to advance the field of chemistry remains unclear. In this paper, rather than pursuing state-of-the-art performance, we aim to evaluate capabilities of LLMs in a wide range of tasks across the chemistry domain. We identify three key chemistry-related capabilities including understanding, reasoning and explaining to explore in LLMs and establish a benchmark containing eight chemistry tasks. Our analysis draws on widely recognized datasets facilitating a broad exploration of the capacities of LLMs within the context of practical chemistry. Five LLMs (GPT-4, GPT-3.5, Davinci-003, Llama and Galactica) are evaluated for each chemistry task in zero-shot and few-shot in-context learning settings with carefully selected demonstration examples and specially crafted prompts. Our investigation found that GPT-4 outperformed other models and LLMs exhibit different competitive levels in eight chemistry tasks. In addition to the key findings from the comprehensive benchmark analysis, our work provides insights into the limitation of current LLMs and the impact of in-context learning settings on LLMs' performance across various chemistry tasks. The code and datasets used in this study are available at this https URL
Submission history
From: Taicheng Guo [view email][v1] Sat, 27 May 2023 14:17:33 GMT (6493kb,D)
[v2] Sun, 10 Sep 2023 16:37:36 GMT (7863kb,D)
[v3] Thu, 28 Dec 2023 04:29:36 GMT (7865kb,D)
Link back to: arXiv, form interface, contact.