We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat.ML

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Machine Learning

Title: Variational Bayes In Private Settings (VIPS)

Abstract: We provide a general framework for privacy-preserving variational Bayes (VB) for a large class of probabilistic models, called the conjugate exponential (CE) family. Our primary observation is that when models are in the CE family, we can privatise the variational posterior distributions simply by perturbing the expected sufficient statistics of the complete-data likelihood. For widely used non-CE models with binomial likelihoods (e.g., logistic regression), we exploit the P{\'o}lya-Gamma data augmentation scheme to bring such models into the CE family, such that inferences in the modified model resemble the original (private) variational Bayes algorithm as closely as possible. The iterative nature of variational Bayes presents a further challenge for privacy preservation, as each iteration increases the amount of noise needed. We overcome this challenge by combining: (1) a relaxed notion of differential privacy, called {\it{concentrated differential privacy}}, which provides a tight bound on the privacy cost of multiple VB iterations and thus significantly decreases the amount of additive noise; and (2) the privacy amplification effect resulting from subsampling mini-batches from large-scale data in stochastic learning. We empirically demonstrate the effectiveness of our method in CE and non-CE models including latent Dirichlet allocation (LDA) and Bayesian logistic regression, evaluated on real-world datasets.
Subjects: Machine Learning (stat.ML); Cryptography and Security (cs.CR)
Cite as: arXiv:1611.00340 [stat.ML]
  (or arXiv:1611.00340v1 [stat.ML] for this version)

Submission history

From: Mijung Park [view email]
[v1] Tue, 1 Nov 2016 19:19:49 GMT (4278kb,D)
[v2] Mon, 28 Nov 2016 21:09:16 GMT (5297kb,D)
[v3] Wed, 21 Dec 2016 23:16:31 GMT (5305kb,D)
[v4] Tue, 6 Mar 2018 20:52:01 GMT (2735kb,D)
[v5] Mon, 3 Dec 2018 20:32:56 GMT (568kb,D)

Link back to: arXiv, form interface, contact.