References & Citations
Computer Science > Machine Learning
Title: Weight Divergence Driven Divide-and-Conquer Approach for Optimal Federated Learning from non-IID Data
(Submitted on 28 Jun 2021 (v1), last revised 30 Jun 2021 (this version, v2))
Abstract: Federated Learning allows training of data stored in distributed devices without the need for centralizing training data, thereby maintaining data privacy. Addressing the ability to handle data heterogeneity (non-identical and independent distribution or non-IID) is a key enabler for the wider deployment of Federated Learning. In this paper, we propose a novel Divide-and-Conquer training methodology that enables the use of the popular FedAvg aggregation algorithm by overcoming the acknowledged FedAvg limitations in non-IID environments. We propose a novel use of Cosine-distance based Weight Divergence metric to determine the exact point where a Deep Learning network can be divided into class agnostic initial layers and class-specific deep layers for performing a Divide and Conquer training. We show that the methodology achieves trained model accuracy at par (and in certain cases exceeding) with numbers achieved by state-of-the-art Aggregation algorithms like FedProx, FedMA, etc. Also, we show that this methodology leads to compute and bandwidth optimizations under certain documented conditions.
Submission history
From: Pravin Chandran [view email][v1] Mon, 28 Jun 2021 09:34:20 GMT (296kb,D)
[v2] Wed, 30 Jun 2021 02:35:38 GMT (296kb,D)
Link back to: arXiv, form interface, contact.