Current browse context:
stat.ML
Change to browse by:
References & Citations
Statistics > Machine Learning
Title: The $χ$-Divergence for Approximate Inference
(Submitted on 1 Nov 2016 (this version), latest version 12 Nov 2017 (v4))
Abstract: Variational inference enables Bayesian analysis for complex probabilistic models with massive data sets. It works by positing a family of distributions and finding the member in the family that is closest to the posterior. While successful, variational methods can run into pathologies; for example, they typically underestimate posterior uncertainty. We propose CHI-VI, a complementary algorithm to traditional variational inference with KL($q$ || $p$) and an alternative algorithm to EP. CHI-VI is a black box algorithm that minimizes the $\chi$-divergence from the posterior to the family of approximating distributions. In EP, only local minimization of the KL($p$ || $q$) objective is possible. In contrast, CHI-VI optimizes a well-defined global objective. It directly minimizes an upper bound to the model evidence that equivalently minimizes the $\chi$-divergence. In experiments, we illustrate the utility of the upper bound for sandwich estimating the model evidence. We also compare several probabilistic models and a Cox process for basketball data. We find CHI-VI often yields better classification error rates and better posterior uncertainty.
Submission history
From: Adji Bousso Dieng [view email][v1] Tue, 1 Nov 2016 18:40:23 GMT (6045kb,D)
[v2] Mon, 27 Feb 2017 03:00:03 GMT (5663kb,D)
[v3] Mon, 6 Nov 2017 00:29:21 GMT (3841kb,D)
[v4] Sun, 12 Nov 2017 19:00:57 GMT (7729kb,D)
Link back to: arXiv, form interface, contact.