Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Privacy-Preserving Federated Learning via Normalized (instead of Clipped) Updates
(Submitted on 13 Jun 2021 (v1), revised 24 Oct 2021 (this version, v2), latest version 16 Apr 2022 (v3))
Abstract: Differentially private federated learning (FL) entails bounding the sensitivity to each client's update. The customary approach used in practice for bounding sensitivity is to \textit{clip} the client updates, which is just projection onto an $\ell_2$ ball of some radius (called the clipping threshold) centered at the origin. However, clipping introduces bias depending on the clipping threshold and its impact on convergence has not been properly analyzed in the FL literature. In this work, we propose a simpler alternative for bounding sensitivity which is \textit{normalization}, i.e. use only the \textit{unit vector} along the client updates, completely discarding the magnitude information. We call this algorithm \texttt{DP-NormFedAvg} and show that it has the same order-wise convergence rate as \texttt{FedAvg} on smooth quasar-convex functions (an important class of non-convex functions for modeling optimization of deep neural networks) modulo the noise variance term (due to privacy). Further, assuming that the per-sample client losses obey a strong-growth type of condition, we show that with high probability, the sensitivity reduces by a factor of $\mathcal{O}(\frac{1}{m})$, where $m$ is the minimum number of samples within a client, compared to its worst-case value. Using this high probability sensitivity value enables us to reduce the iteration complexity of \texttt{DP-NormFedAvg} by a factor of $\mathcal{O}(\frac{1}{m^2})$, at the expense of an exponentially small degradation in the privacy guarantee. We also corroborate our theory with experiments on neural networks.
Submission history
From: Abolfazl Hashemi [view email][v1] Sun, 13 Jun 2021 21:23:46 GMT (629kb,D)
[v2] Sun, 24 Oct 2021 15:22:47 GMT (213kb,D)
[v3] Sat, 16 Apr 2022 03:16:33 GMT (484kb,D)
Link back to: arXiv, form interface, contact.