Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Collaborative Learning in the Jungle (Decentralized, Byzantine, Heterogeneous, Asynchronous and Nonconvex Learning)
(Submitted on 3 Aug 2020 (v1), last revised 1 Dec 2021 (this version, v5))
Abstract: We study Byzantine collaborative learning, where $n$ nodes seek to collectively learn from each others' local data. The data distribution may vary from one node to another. No node is trusted, and $f < n$ nodes can behave arbitrarily. We prove that collaborative learning is equivalent to a new form of agreement, which we call averaging agreement. In this problem, nodes start each with an initial vector and seek to approximately agree on a common vector, which is close to the average of honest nodes' initial vectors. We present two asynchronous solutions to averaging agreement, each we prove optimal according to some dimension. The first, based on the minimum-diameter averaging, requires $ n \geq 6f+1$, but achieves asymptotically the best-possible averaging constant up to a multiplicative constant. The second, based on reliable broadcast and coordinate-wise trimmed mean, achieves optimal Byzantine resilience, i.e., $n \geq 3f+1$. Each of these algorithms induces an optimal Byzantine collaborative learning protocol. In particular, our equivalence yields new impossibility theorems on what any collaborative learning algorithm can achieve in adversarial and heterogeneous environments.
Submission history
From: Lê-Nguyên Hoang [view email][v1] Mon, 3 Aug 2020 09:44:07 GMT (59kb)
[v2] Tue, 4 Aug 2020 09:56:27 GMT (502kb)
[v3] Mon, 14 Dec 2020 16:30:24 GMT (538kb)
[v4] Mon, 7 Jun 2021 15:05:52 GMT (176kb,D)
[v5] Wed, 1 Dec 2021 15:54:22 GMT (291kb,D)
Link back to: arXiv, form interface, contact.