We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Revisiting Model-Agnostic Private Learning: Faster Rates and Active Learning

Abstract: The Private Aggregation of Teacher Ensembles (PATE) framework is one of the most promising recent approaches in differentially private learning. Existing theoretical analysis shows that PATE consistently learns any VC-classes in the realizable setting, but falls short in explaining its success in more general cases where the error rate of the optimal classifier is bounded away from zero. We fill in this gap by introducing the Tsybakov Noise Condition (TNC) and establish stronger and more interpretable learning bounds. These bounds provide new insights into when PATE works and improve over existing results even in the narrower realizable setting. We also investigate the compelling idea of using active learning for saving privacy budget, and empirical studies show the effectiveness of this new idea. The novel components in the proofs include a more refined analysis of the majority voting classifier -- which could be of independent interest -- and an observation that the synthetic "student" learning problem is nearly realizable by construction under the Tsybakov noise condition.
Comments: The paper was published in AISTATS-2021 and the longer version of the paper is under review at JMLR
Subjects: Machine Learning (cs.LG); Cryptography and Security (cs.CR)
Cite as: arXiv:2011.03186 [cs.LG]
  (or arXiv:2011.03186v3 [cs.LG] for this version)

Submission history

From: Chong Liu [view email]
[v1] Fri, 6 Nov 2020 04:35:32 GMT (1015kb,D)
[v2] Fri, 13 Nov 2020 08:19:15 GMT (259kb,D)
[v3] Tue, 21 Sep 2021 18:02:38 GMT (252kb,D)

Link back to: arXiv, form interface, contact.