Current browse context:
cs.DC
Change to browse by:
References & Citations
Computer Science > Distributed, Parallel, and Cluster Computing
Title: A Quantitative Survey of Communication Optimizations in Distributed Deep Learning
(Submitted on 27 May 2020 (v1), last revised 7 Nov 2020 (this version, v2))
Abstract: Nowadays, large and complex deep learning (DL) models are increasingly trained in a distributed manner across multiple worker machines, in which extensive communications between workers pose serious scaling problems. In this article, we present a quantitative survey of communication optimization techniques for data parallel distributed DL. We first identify the major communication challenges and classify the existing solutions into three levels, namely the learning algorithm, the system architecture, and the network infrastructure. We present the state-of-the-art communication optimization techniques and conduct a comparative study of seven common lossless distributed DL methods on a 32-GPU cluster with 100Gbps InfiniBand (IB). We show that (1) the DL models with low model intensity (such as BERT and BERT-Large) are difficult to scale out even with the best available lossless algorithm over 100Gbps IB; (2) the system architecture and scheduling algorithms have a critical impact on the scaling property. We conclude the article with discussions on the open issues for further investigations.
Submission history
From: Shaohuai Shi [view email][v1] Wed, 27 May 2020 09:12:48 GMT (2571kb,D)
[v2] Sat, 7 Nov 2020 07:05:33 GMT (2364kb,D)
Link back to: arXiv, form interface, contact.