Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Optimizing the Communication-Accuracy Trade-off in Federated Learning with Rate-Distortion Theory
(Submitted on 7 Jan 2022 (v1), last revised 19 May 2022 (this version, v3))
Abstract: A significant bottleneck in federated learning (FL) is the network communication cost of sending model updates from client devices to the central server. We present a comprehensive empirical study of the statistics of model updates in FL, as well as the role and benefits of various compression techniques. Motivated by these observations, we propose a novel method to reduce the average communication cost, which is near-optimal in many use cases, and outperforms Top-K, DRIVE, 3LC and QSGD on Stack Overflow next-word prediction, a realistic and challenging FL benchmark. This is achieved by examining the problem using rate-distortion theory, and proposing distortion as a reliable proxy for model accuracy. Distortion can be more effectively used for optimizing the trade-off between model performance and communication cost across clients. We demonstrate empirically that in spite of the non-i.i.d. nature of federated learning, the rate-distortion frontier is consistent across datasets, optimizers, clients and training rounds.
Submission history
From: Nicole Mitchell [view email][v1] Fri, 7 Jan 2022 20:17:33 GMT (1732kb,D)
[v2] Tue, 15 Mar 2022 16:45:09 GMT (1883kb,D)
[v3] Thu, 19 May 2022 18:18:32 GMT (4110kb,D)
Link back to: arXiv, form interface, contact.