Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Using Large Ensembles of Control Variates for Variational Inference
(Submitted on 30 Oct 2018 (v1), last revised 22 Oct 2020 (this version, v2))
Abstract: Variational inference is increasingly being addressed with stochastic optimization. In this setting, the gradient's variance plays a crucial role in the optimization procedure, since high variance gradients lead to poor convergence. A popular approach used to reduce gradient's variance involves the use of control variates. Despite the good results obtained, control variates developed for variational inference are typically looked at in isolation. In this paper we clarify the large number of control variates that are available by giving a systematic view of how they are derived. We also present a Bayesian risk minimization framework in which the quality of a procedure for combining control variates is quantified by its effect on optimization convergence rates, which leads to a very simple combination rule. Results show that combining a large number of control variates this way significantly improves the convergence of inference over using the typical gradient estimators or a reduced number of control variates.
Submission history
From: Tomas Geffner [view email][v1] Tue, 30 Oct 2018 01:48:19 GMT (304kb,D)
[v2] Thu, 22 Oct 2020 11:03:48 GMT (643kb,D)
Link back to: arXiv, form interface, contact.