We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Bringing Differential Private SGD to Practice: On the Independence of Gaussian Noise and the Number of Training Rounds

Abstract: In the context of DP-SGD each round communicates a local SGD update which leaks some new information about the underlying local data set to the outside world. In order to provide privacy, Gaussian noise is added to local SGD updates. However, privacy leakage still aggregates over multiple training rounds. Therefore, in order to control privacy leakage over an increasing number of training rounds, we need to increase the added Gaussian noise per local SGD update. This dependence of the amount of Gaussian noise $\sigma$ on the number of training rounds $T$ may impose an impractical upper bound on $T$ (because $\sigma$ cannot be too large) leading to a low accuracy global model (because the global model receives too few local SGD updates). DP-SGD much less competitive compared to other existing privacy techniques.
We show for the first time that for $(\epsilon,\delta)$-differential privacy $\sigma$ can be chosen equal to $\sqrt{2(\epsilon +\ln(1/\delta))/\epsilon}$ regardless the total number of training rounds $T$. In other words, $\sigma$ does not depend on $T$ anymore (and aggregation of privacy leakage increases to a limit). This important discovery brings DP-SGD to practice because $\sigma$ can remain small to make the trained model have high accuracy even for large $T$ as usually happens in practice.
Comments: arXiv admin note: text overlap with arXiv:2007.09208
Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC); Machine Learning (stat.ML)
Cite as: arXiv:2102.09030 [cs.LG]
  (or arXiv:2102.09030v2 [cs.LG] for this version)

Submission history

From: Ngoc Toan Nguyen [view email]
[v1] Wed, 17 Feb 2021 21:19:39 GMT (18148kb,D)
[v2] Mon, 14 Jun 2021 21:05:08 GMT (1727kb,D)
[v3] Tue, 19 Oct 2021 23:07:08 GMT (1755kb,D)
[v4] Tue, 1 Feb 2022 00:27:02 GMT (18773kb,D)

Link back to: arXiv, form interface, contact.