We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Preserved central model for faster bidirectional compression in distributed settings

Abstract: We develop a new approach to tackle communication constraints in a distributed learning problem with a central server. We propose and analyze a new algorithm that performs bidirectional compression and achieves the same convergence rate as algorithms using only uplink (from the local workers to the central server) compression. To obtain this improvement, we design MCM, an algorithm such that the downlink compression only impacts local models, while the global model is preserved. As a result, and contrary to previous works, the gradients on local servers are computed on perturbed models. Consequently, convergence proofs are more challenging and require a precise control of this perturbation. To ensure it, MCM additionally combines model compression with a memory mechanism. This analysis opens new doors, e.g. incorporating worker dependent randomized-models and partial participation.
Comments: Bidirectional compression for Federated Learning: 59 pages, 5 theorems, published at NeurIPS 2021
Subjects: Machine Learning (cs.LG); Distributed, Parallel, and Cluster Computing (cs.DC); Statistics Theory (math.ST)
Journal reference: Advances in Neural Information Processing Systems, 2021, vol. 34, p. 2387-2399
Cite as: arXiv:2102.12528 [cs.LG]
  (or arXiv:2102.12528v2 [cs.LG] for this version)

Submission history

From: Constantin Philippenko [view email]
[v1] Wed, 24 Feb 2021 19:48:20 GMT (3076kb,D)
[v2] Thu, 16 Jun 2022 14:00:00 GMT (4622kb)

Link back to: arXiv, form interface, contact.