We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat.ML

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Machine Learning

Title: Learning Energy-Based Model with Variational Auto-Encoder as Amortized Sampler

Abstract: Due to the intractable partition function, training energy-based models (EBMs) by maximum likelihood requires Markov chain Monte Carlo (MCMC) sampling to approximate the gradient of the Kullback-Leibler divergence between data and model distributions. However, it is non-trivial to sample from an EBM because of the difficulty of mixing between modes. In this paper, we propose to learn a variational auto-encoder (VAE) to initialize the finite-step MCMC, such as Langevin dynamics that is derived from the energy function, for efficient amortized sampling of the EBM. With these amortized MCMC samples, the EBM can be trained by maximum likelihood, which follows an "analysis by synthesis" scheme; while the VAE learns from these MCMC samples via variational Bayes. We call this joint training algorithm the variational MCMC teaching, in which the VAE chases the EBM toward data distribution. We interpret the learning algorithm as a dynamic alternating projection in the context of information geometry. Our proposed models can generate samples comparable to GANs and EBMs. Additionally, we demonstrate that our model can learn effective probabilistic distribution toward supervised conditional learning tasks.
Subjects: Machine Learning (stat.ML); Machine Learning (cs.LG)
Journal reference: Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI), pp10441--10451, 2021
Cite as: arXiv:2012.14936 [stat.ML]
  (or arXiv:2012.14936v2 [stat.ML] for this version)

Submission history

From: Jianwen Xie [view email]
[v1] Tue, 29 Dec 2020 20:46:40 GMT (3214kb,D)
[v2] Fri, 24 Dec 2021 00:23:52 GMT (3414kb,D)

Link back to: arXiv, form interface, contact.