Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Bounds
(Submitted on 24 Jun 2020 (v1), last revised 10 Aug 2020 (this version, v2))
Abstract: We study differentially private (DP) algorithms for stochastic non-convex optimization. In this problem, the goal is to minimize the population loss over a $p$-dimensional space given $n$ i.i.d. samples drawn from a distribution. We improve upon the population gradient bound of ${\sqrt{p}}/{\sqrt{n}}$ from prior work and obtain a sharper rate of $\sqrt[4]{p}/\sqrt{n}$. We obtain this rate by providing the first analyses on a collection of private gradient-based methods, including adaptive algorithms DP RMSProp and DP Adam. Our proof technique leverages the connection between differential privacy and adaptive data analysis to bound gradient estimation error at every iterate, which circumvents the worse generalization bound from the standard uniform convergence argument. Finally, we evaluate the proposed algorithms on two popular deep learning tasks and demonstrate the empirical advantages of DP adaptive gradient methods over standard DP SGD.
Submission history
From: Yingxue Zhou [view email][v1] Wed, 24 Jun 2020 06:01:24 GMT (951kb,D)
[v2] Mon, 10 Aug 2020 20:25:35 GMT (472kb,D)
Link back to: arXiv, form interface, contact.