Current browse context:
cs.DC
Change to browse by:
References & Citations
Computer Science > Distributed, Parallel, and Cluster Computing
Title: Elixir: Train a Large Language Model on a Small GPU Cluster
(Submitted on 10 Dec 2022 (v1), last revised 31 May 2023 (this version, v3))
Abstract: In recent years, large language models have achieved great success due to their unprecedented size. However, training these models poses a challenge for most researchers as it requires a substantial number of GPUs. To reduce GPU memory usage, memory partitioning, and memory offloading have been proposed. These approaches eliminate memory redundancies and offload memory usage to the CPU and NVMe memory, respectively, enabling training on small GPU clusters. However, directly deploying these solutions often leads to suboptimal efficiency. Only experienced experts can unleash the full potential of hardware by carefully tuning the distributed configuration. Thus, we present a novel solution, Elixir, which automates efficient large-model training based on pre-runtime model profiling. Elixir aims to identify the optimal combination of partitioning and offloading techniques to maximize training throughput. In our experiments, Elixir significantly outperforms the current state-of-the-art baseline. Our optimal configuration achieves up to a 3.4$\times$ speedup on GPT-2 models compared with SOTA solutions. We hope that our work will benefit individuals who lack computing resources and expertise, granting them access to large models. The beta version of Elixir is now available at this https URL
Submission history
From: Yang You [view email][v1] Sat, 10 Dec 2022 17:26:05 GMT (1237kb,D)
[v2] Sun, 26 Feb 2023 14:38:09 GMT (2652kb,D)
[v3] Wed, 31 May 2023 13:56:53 GMT (1333kb,D)
Link back to: arXiv, form interface, contact.