We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.DC

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: CD-SGD: Distributed Stochastic Gradient Descent with Compression and Delay Compensation

Abstract: Communication overhead is the key challenge for distributed training. Gradient compression is a widely used approach to reduce communication traffic. When combining with parallel communication mechanism method like pipeline, gradient compression technique can greatly alleviate the impact of communication overhead. However, there exists two problems of gradient compression technique to be solved. Firstly, gradient compression brings in extra computation cost, which will delay the next training iteration. Secondly, gradient compression usually leads to the decrease of convergence accuracy.
Comments: 12 pages
Subjects: Machine Learning (cs.LG); Distributed, Parallel, and Cluster Computing (cs.DC)
Cite as: arXiv:2106.10796 [cs.LG]
  (or arXiv:2106.10796v2 [cs.LG] for this version)

Submission history

From: Enda Yu [view email]
[v1] Mon, 21 Jun 2021 01:15:12 GMT (5633kb,D)
[v2] Tue, 7 Sep 2021 01:44:00 GMT (5634kb,D)

Link back to: arXiv, form interface, contact.