We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Distributed, Parallel, and Cluster Computing

Title: Distributed Deep Learning Using Volunteer Computing-Like Paradigm

Abstract: Use of Deep Learning (DL) in commercial applications such as image classification, sentiment analysis and speech recognition is increasing. When training DL models with large number of parameters and/or large datasets, cost and speed of training can become prohibitive. Distributed DL training solutions that split a training job into subtasks and execute them over multiple nodes can decrease training time. However, the cost of current solutions, built predominantly for cluster computing systems, can still be an issue. In contrast to cluster computing systems, Volunteer Computing (VC) systems can lower the cost of computing, but applications running on VC systems have to handle fault tolerance, variable network latency and heterogeneity of compute nodes, and the current solutions are not designed to do so. We design a distributed solution that can run DL training on a VC system by using a data parallel approach. We implement a novel asynchronous SGD scheme called VC-ASGD suited for VC systems. In contrast to traditional VC systems that lower cost by using untrustworthy volunteer devices, we lower cost by leveraging preemptible computing instances on commercial cloud platforms. By using preemptible instances that require applications to be fault tolerant, we lower cost by 70-90% and improve data security.
Subjects: Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
ACM classes: C.2.4; I.2.11
Journal reference: ScaDL workshop at IEEE International Parallel and Distributed Processing Symposium 2021
DOI: 10.1109/IPDPSW52791.2021.00144
Cite as: arXiv:2103.08894 [cs.DC]
  (or arXiv:2103.08894v3 [cs.DC] for this version)

Submission history

From: Medha Atre [view email]
[v1] Tue, 16 Mar 2021 07:32:58 GMT (306kb,D)
[v2] Fri, 2 Apr 2021 12:50:05 GMT (768kb,D)
[v3] Thu, 27 May 2021 06:41:45 GMT (768kb,D)

Link back to: arXiv, form interface, contact.