We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: IntSGD: Adaptive Floatless Compression of Stochastic Gradients

Abstract: We propose a family of adaptive integer compression operators for distributed Stochastic Gradient Descent (SGD) that do not communicate a single float. This is achieved by multiplying floating-point vectors with a number known to every device and then rounding to integers. In contrast to the prior work on integer compression for SwitchML by Sapio et al. (2021), our IntSGD method is provably convergent and computationally cheaper as it estimates the scaling of vectors adaptively. Our theory shows that the iteration complexity of IntSGD matches that of SGD up to constant factors for both convex and non-convex, smooth and non-smooth functions, with and without overparameterization. Moreover, our algorithm can also be tailored for the popular all-reduce primitive and shows promising empirical performance.
Comments: Spotlight at ICLR 2022. 27 pages, 6 figures, 3 algorithms
Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC); Machine Learning (stat.ML)
Journal reference: International Conference on Learning Representations (2022)
Cite as: arXiv:2102.08374 [cs.LG]
  (or arXiv:2102.08374v2 [cs.LG] for this version)

Submission history

From: Konstantin Mishchenko [view email]
[v1] Tue, 16 Feb 2021 18:58:57 GMT (4776kb,D)
[v2] Sun, 20 Mar 2022 23:21:09 GMT (20778kb,D)

Link back to: arXiv, form interface, contact.