We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networks

Abstract: Distributed data-parallel training has been widely adopted for deep neural network (DNN) models. Although current deep learning (DL) frameworks scale well for dense models like image classification models, we find that these DL frameworks have relatively low scalability for sparse models like natural language processing (NLP) models that have highly sparse embedding tables. Most existing works overlook the sparsity of model parameters thus suffering from significant but unnecessary communication overhead. In this paper, we propose EmbRace, an efficient communication framework to accelerate communications of distributed training for sparse models. EmbRace introduces Sparsity-aware Hybrid Communication, which integrates AlltoAll and model parallelism into data-parallel training, so as to reduce the communication overhead of highly sparse parameters. To effectively overlap sparse communication with both backward and forward computation, EmbRace further designs a 2D Communication Scheduling approach which optimizes the model computation procedure, relaxes the dependency of embeddings, and schedules the sparse communications of each embedding row with a priority queue. We have implemented a prototype of EmbRace based on PyTorch and Horovod, and conducted comprehensive evaluations with four representative NLP models. Experimental results show that EmbRace achieves up to 2.41X speedup compared to the state-of-the-art distributed training baselines.
Subjects: Machine Learning (cs.LG); Multiagent Systems (cs.MA)
DOI: 10.1145/3545008.3545011
Cite as: arXiv:2110.09132 [cs.LG]
  (or arXiv:2110.09132v2 [cs.LG] for this version)

Submission history

From: Zhiquan Lai [view email]
[v1] Mon, 18 Oct 2021 09:35:40 GMT (7994kb,D)
[v2] Mon, 27 Jun 2022 03:10:55 GMT (4992kb,D)

Link back to: arXiv, form interface, contact.