Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Communication-Efficient and Byzantine-Robust Distributed Learning
(Submitted on 21 Nov 2019 (v1), revised 10 May 2020 (this version, v2), latest version 14 Aug 2021 (v5))
Abstract: We develop a communication-efficient distributed learning algorithm that is robust against Byzantine worker machines. We propose and analyze a distributed gradient-descent algorithm that performs a simple thresholding based on gradient norms to mitigate Byzantine failures. We show the (statistical) error-rate of our algorithm matches that of [YCKB18], which uses more complicated schemes (like coordinate-wise median or trimmed mean) and thus optimal. Furthermore, for communication efficiency, we consider a generic class of {\delta}-approximate compressors from [KRSJ19] that encompasses sign-based compressors and top-k sparsification. Our algorithm uses compressed gradients and gradient norms for aggregation and Byzantine removal respectively. We establish the statistical error rate of the algorithm for arbitrary (convex or non-convex) smooth loss function. We show that, in the regime when the compression factor {\delta} is constant and the dimension of the parameter space is fixed, the rate of convergence is not affected by the compression operation, and hence we effectively get the compression for free. Moreover, we extend the compressed gradient descent algorithm with error feedback proposed in [KRSJ19] for the distributed setting. We have experimentally validated our results and shown good performance in convergence for convex (least-square regression) and non-convex (neural network training) problems.
Submission history
From: Raj Kumar Maity [view email][v1] Thu, 21 Nov 2019 19:39:53 GMT (1693kb,D)
[v2] Sun, 10 May 2020 20:04:58 GMT (11252kb,D)
[v3] Fri, 10 Jul 2020 20:27:47 GMT (11252kb,D)
[v4] Thu, 11 Mar 2021 20:21:26 GMT (3375kb,D)
[v5] Sat, 14 Aug 2021 21:21:41 GMT (2422kb,D)
Link back to: arXiv, form interface, contact.