We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Machine Learning

Title: Tightening Bounds for Variational Inference by Revisiting Perturbation Theory

Abstract: Variational inference has become one of the most widely used methods in latent variable modeling. In its basic form, variational inference employs a fully factorized variational distribution and minimizes its KL divergence to the posterior. As the minimization can only be carried out approximately, this approximation induces a bias. In this paper, we revisit perturbation theory as a powerful way of improving the variational approximation. Perturbation theory relies on a form of Taylor expansion of the log marginal likelihood, vaguely in terms of the log ratio of the true posterior and its variational approximation. While first order terms give the classical variational bound, higher-order terms yield corrections that tighten it. However, traditional perturbation theory does not provide a lower bound, making it inapt for stochastic optimization. In this paper, we present a similar yet alternative way of deriving corrections to the ELBO that resemble perturbation theory, but that result in a valid bound. We show in experiments on Gaussian Processes and Variational Autoencoders that the new bounds are more mass covering, and that the resulting posterior covariances are closer to the true posterior and lead to higher likelihoods on held-out data.
Comments: To appear in Journal of Statistical Mechanics: Theory and Experiment (JSTAT), 2019
Subjects: Machine Learning (stat.ML); Machine Learning (cs.LG)
DOI: 10.1088/1742-5468/ab43d3
Cite as: arXiv:1910.00069 [stat.ML]
  (or arXiv:1910.00069v1 [stat.ML] for this version)

Submission history

From: Robert Bamler [view email]
[v1] Mon, 30 Sep 2019 19:44:18 GMT (295kb)

Link back to: arXiv, form interface, contact.