References & Citations
Statistics > Machine Learning
Title: Extreme Stochastic Variational Inference: Distributed and Asynchronous
(Submitted on 31 May 2016 (v1), revised 15 Feb 2018 (this version, v6), latest version 3 Aug 2018 (v9))
Abstract: We propose extreme stochastic variational inference (ESVI), an asynchronous and lock-free algorithm to perform variational inference on massive real world datasets. Stochastic variational inference (SVI), the state-of-the-art algorithm for scaling variational inference to large-datasets, is inherently serial. Moreover, it requires the parameters to fit in the memory of a single processor; this is problematic when the number of parameters is in billions. ESVI overcomes these limitations by requiring that each processor only access a subset of the data and a subset of the parameters, thus providing data and model parallelism simultaneously. We demonstrate the effectiveness of ESVI by running Latent Dirichlet Allocation (LDA) on UMBC-3B, a dataset that has a vocabulary of 3 million and a token size of 3 billion. To best of our knowledge, this is an order of magnitude larger than the largest dataset on which results using variational inference have been reported in literature. In our experiments, we found that ESVI outperforms VI and SVI, and also achieves a better quality solution. In addition, we propose a strategy to speed up computation and save memory when fitting large number of topics.
Submission history
From: Parameswaran Raman [view email][v1] Tue, 31 May 2016 05:10:51 GMT (2298kb,D)
[v2] Wed, 22 Jun 2016 05:55:09 GMT (2298kb,D)
[v3] Wed, 15 Mar 2017 20:50:37 GMT (2581kb,D)
[v4] Fri, 21 Apr 2017 17:37:39 GMT (2692kb,D)
[v5] Tue, 23 May 2017 08:14:27 GMT (1571kb,D)
[v6] Thu, 15 Feb 2018 01:02:49 GMT (904kb,D)
[v7] Wed, 18 Apr 2018 01:10:33 GMT (907kb,D)
[v8] Mon, 21 May 2018 23:39:58 GMT (907kb,D)
[v9] Fri, 3 Aug 2018 22:05:49 GMT (905kb,D)
Link back to: arXiv, form interface, contact.