We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Partitioning sparse deep neural networks for scalable training and inference

Abstract: The state-of-the-art deep neural networks (DNNs) have significant computational and data management requirements. The size of both training data and models continue to increase. Sparsification and pruning methods are shown to be effective in removing a large fraction of connections in DNNs. The resulting sparse networks present unique challenges to further improve the computational efficiency of training and inference in deep learning. Both the feedforward (inference) and backpropagation steps in stochastic gradient descent (SGD) algorithm for training sparse DNNs involve consecutive sparse matrix-vector multiplications (SpMVs). We first introduce a distributed-memory parallel SpMV-based solution for the SGD algorithm to improve its scalability. The parallelization approach is based on row-wise partitioning of weight matrices that represent neuron connections between consecutive layers. We then propose a novel hypergraph model for partitioning weight matrices to reduce the total communication volume and ensure computational load-balance among processors. Experiments performed on sparse DNNs demonstrate that the proposed solution is highly efficient and scalable. By utilizing the proposed matrix partitioning scheme, the performance of our solution is further improved significantly.
Comments: Gunduz Vehbi Demirci and Hakan Ferhatosmanoglu. 2021. Partitioning Sparse Deep Neural Networks for Scalable Training and Inference. In 2021 International Conference on Supercomputing (ICS '21), June 14-17, 2021, Virtual Event, USA. ACM, New York, NY, USA, 12 pages
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC)
DOI: 10.1145/3447818.3460372
Cite as: arXiv:2104.11805 [cs.LG]
  (or arXiv:2104.11805v1 [cs.LG] for this version)

Submission history

From: Gunduz Vehbi Demirci [view email]
[v1] Fri, 23 Apr 2021 20:05:52 GMT (505kb,D)

Link back to: arXiv, form interface, contact.