We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization

Abstract: As the size and complexity of models and datasets grow, so does the need for communication-efficient variants of stochastic gradient descent that can be deployed to perform parallel model training. One popular communication-compression method for data-parallel SGD is QSGD (Alistarh et al., 2017), which quantizes and encodes gradients to reduce communication costs. The baseline variant of QSGD provides strong theoretical guarantees, however, for practical purposes, the authors proposed a heuristic variant which we call QSGDinf, which demonstrated impressive empirical gains for distributed training of large neural networks. In this paper, we build on this work to propose a new gradient quantization scheme, and show that it has both stronger theoretical guarantees than QSGD, and matches and exceeds the empirical performance of the QSGDinf heuristic and of other compression methods.
Comments: This entry is redundant and was created in error. See arXiv:1908.06077 for the latest version
Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC); Machine Learning (stat.ML)
Cite as: arXiv:2104.13818 [cs.LG]
  (or arXiv:2104.13818v2 [cs.LG] for this version)

Submission history

From: Ali Ramezani-Kebrya [view email]
[v1] Wed, 28 Apr 2021 15:07:03 GMT (10253kb,D)
[v2] Sat, 1 May 2021 20:34:38 GMT (0kb,I)

Link back to: arXiv, form interface, contact.