We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.DC

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Distributed, Parallel, and Cluster Computing

Title: MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloud

Abstract: Existing general purpose frameworks for gigantic model training, i.e., dense models with billions of parameters, cannot scale efficiently on cloud environment with various networking conditions due to large communication overheads. In this paper, we propose MiCS, which Minimizes the Communication Scale to bring down communication overhead. Specifically, by decreasing the number of participants in a communication collective, MiCS can utilize heterogeneous network bandwidth, reduce network traffic over slower links, reduce the latency of communications for maintaining high network bandwidth utilization, and amortize expensive global gradient synchronization overhead. Our evaluation on AWS shows that the system throughput of MiCS is up to 2.89$\times$ that of the state-of-the-art large model training systems. MiCS achieves near-linear scaling efficiency, which is up to 1.27$\times$ that of DeepSpeed. MiCS allows us to train a proprietary model with 100 billion parameters on 512 GPUs with 99.4% weak-scaling efficiency, and it is able to saturate over 54.5% theoretical computation power of each GPU on a public cloud with less GPU memory and more restricted networks than DGX-A100 clusters.
Subjects: Distributed, Parallel, and Cluster Computing (cs.DC)
Cite as: arXiv:2205.00119 [cs.DC]
  (or arXiv:2205.00119v5 [cs.DC] for this version)

Submission history

From: Zhen Zhang [view email]
[v1] Sat, 30 Apr 2022 00:55:29 GMT (2636kb,D)
[v2] Tue, 3 May 2022 04:45:13 GMT (1883kb,D)
[v3] Wed, 4 May 2022 15:59:35 GMT (1883kb,D)
[v4] Tue, 24 May 2022 19:18:31 GMT (1878kb,D)
[v5] Fri, 28 Oct 2022 15:55:09 GMT (1847kb,D)

Link back to: arXiv, form interface, contact.