Current browse context:
stat.ML
Change to browse by:
References & Citations
Statistics > Machine Learning
Title: Langevin Monte Carlo: random coordinate descent and variance reduction
(Submitted on 26 Jul 2020 (v1), revised 9 Mar 2021 (this version, v4), latest version 7 Oct 2021 (v8))
Abstract: Sampling from a log-concave distribution function on $\mathbb{R}^d$ (with $d\gg 1$) is a popular problem that has wide applications. In this paper we study the application of random coordinate descent method (RCD) on the Langevin Monte Carlo (LMC) sampling method, and we find two sides of the theory:
1. The direct application of RCD on LMC does reduce the number of finite differencing approximations per iteration, but it induces a large variance error term. More iterations are then needed, and ultimately the method gains no computational advantage;
2. When variance reduction techniques (such as SAGA and SVRG) are incorporated in RCD-LMC, the variance error term is reduced. The new methods, compared to the vanilla LMC, reduce the total computational cost by $d$ folds, and achieve the optimal cost rate.
We perform our investigations in both overdamped and underdamped settings.
Submission history
From: Zhiyan Ding [view email][v1] Sun, 26 Jul 2020 18:14:36 GMT (56kb)
[v2] Wed, 29 Jul 2020 20:22:59 GMT (56kb)
[v3] Fri, 12 Feb 2021 17:14:10 GMT (67kb)
[v4] Tue, 9 Mar 2021 23:45:10 GMT (72kb)
[v5] Sun, 4 Jul 2021 00:01:15 GMT (69kb)
[v6] Mon, 13 Sep 2021 16:47:07 GMT (70kb)
[v7] Sun, 3 Oct 2021 18:33:13 GMT (72kb)
[v8] Thu, 7 Oct 2021 14:07:11 GMT (71kb)
Link back to: arXiv, form interface, contact.