References & Citations
Computer Science > Computation and Language
Title: Pretraining and Updating Language- and Domain-specific Large Language Model: A Case Study in Japanese Business Domain
(Submitted on 12 Apr 2024 (this version), latest version 16 Apr 2024 (v2))
Abstract: Several previous studies have considered language- and domain-specific large language models (LLMs) as separate topics. This study explores the combination of a non-English language and a high-demand industry domain, focusing on a Japanese business-specific LLM. This type of a model requires expertise in the business domain, strong language skills, and regular updates of its knowledge. We trained a 13-billion-parameter LLM from scratch using a new dataset of business texts and patents, and continually pretrained it with the latest business documents. Further we propose a new benchmark for Japanese business domain question answering (QA) and evaluate our models on it. The results show that our pretrained model improves QA accuracy without losing general knowledge, and that continual pretraining enhances adaptation to new information. Our pretrained model and business domain benchmark are publicly available.
Submission history
From: Kosuke Takahashi [view email][v1] Fri, 12 Apr 2024 06:21:48 GMT (64kb)
[v2] Tue, 16 Apr 2024 02:24:00 GMT (64kb)
Link back to: arXiv, form interface, contact.