Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: On the Convergence and Calibration of Deep Learning with Differential Privacy
(Submitted on 15 Jun 2021 (v1), last revised 19 Jun 2023 (this version, v6))
Abstract: Differentially private (DP) training preserves the data privacy usually at the cost of slower convergence (and thus lower accuracy), as well as more severe mis-calibration than its non-private counterpart. To analyze the convergence of DP training, we formulate a continuous time analysis through the lens of neural tangent kernel (NTK), which characterizes the per-sample gradient clipping and the noise addition in DP training, for arbitrary network architectures and loss functions. Interestingly, we show that the noise addition only affects the privacy risk but not the convergence or calibration, whereas the per-sample gradient clipping (under both flat and layerwise clipping styles) only affects the convergence and calibration.
Furthermore, we observe that while DP models trained with small clipping norm usually achieve the best accurate, but are poorly calibrated and thus unreliable. In sharp contrast, DP models trained with large clipping norm enjoy the same privacy guarantee and similar accuracy, but are significantly more \textit{calibrated}. Our code can be found at \url{this https URL}.
Submission history
From: Zongyu Dai [view email][v1] Tue, 15 Jun 2021 01:32:29 GMT (1733kb,D)
[v2] Sat, 17 Jul 2021 04:11:06 GMT (2027kb,D)
[v3] Sun, 10 Oct 2021 04:41:31 GMT (1617kb,D)
[v4] Sat, 29 Jan 2022 05:25:22 GMT (1622kb,D)
[v5] Tue, 1 Feb 2022 22:38:40 GMT (1622kb,D)
[v6] Mon, 19 Jun 2023 15:13:37 GMT (1166kb,D)
Link back to: arXiv, form interface, contact.