### Current browse context:

cs.LG

### Change to browse by:

### References & Citations

# Computer Science > Machine Learning

# Title: Bringing Differential Private SGD to Practice: On the Independence of Gaussian Noise and the Number of Training Rounds

(Submitted on 17 Feb 2021 (v1), revised 19 Oct 2021 (this version, v3),

*latest version 1 Feb 2022*(v4))Abstract: In the context of DP-SGD each round communicates a local SGD update which leaks some new information about the underlying local data set to the outside world. In order to provide privacy, Gaussian noise is added to local SGD updates. However, privacy leakage still aggregates over multiple training rounds. Therefore, in order to control privacy leakage over an increasing number of training rounds, we need to increase the added Gaussian noise per local SGD update. This dependence of the amount of Gaussian noise $\sigma$ on the number of training rounds $T$ may impose an impractical upper bound on $T$ (because $\sigma$ cannot be too large) leading to a low accuracy global model (because the global model receives too few local SGD updates). This makes DP-SGD much less competitive compared to other existing privacy techniques.

We show for the first time that for $(\epsilon,\delta)$-differential privacy $\sigma$ can be chosen equal to $\sqrt{2(\epsilon +\ln(1/\delta))/\epsilon}$ for $\epsilon=\Omega(T/N^2)$. In many existing machine learning problems, $N$ is always large and $T=O(N)$. Hence, $\sigma$ becomes ``independent'' of any $T=O(N)$ choice with $\epsilon=\Omega(1/N)$ (aggregation of privacy leakage increases to a limit). This means that our $\sigma$ only depends on $N$ rather than $T$. This important discovery brings DP-SGD to practice -- as also demonstrated by experiments -- because $\sigma$ can remain small to make the trained model have high accuracy even for large $T$ as usually happens in practice.

## Submission history

From: Ngoc Toan Nguyen [view email]**[v1]**Wed, 17 Feb 2021 21:19:39 GMT (18148kb,D)

**[v2]**Mon, 14 Jun 2021 21:05:08 GMT (1727kb,D)

**[v3]**Tue, 19 Oct 2021 23:07:08 GMT (1755kb,D)

**[v4]**Tue, 1 Feb 2022 00:27:02 GMT (18773kb,D)

Link back to: arXiv, form interface, contact.