We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Condensed Matter > Disordered Systems and Neural Networks

Title: Barriers and Dynamical Paths in Alternating Gibbs Sampling of Restricted Boltzmann Machines

Abstract: Restricted Boltzmann Machines (RBM) are bi-layer neural networks used for the unsupervised learning of model distributions from data. The bipartite architecture of RBM naturally defines an elegant sampling procedure, called Alternating Gibbs Sampling (AGS), where the configurations of the latent-variable layer are sampled conditional to the data-variable layer, and vice versa. We study here the performance of AGS on several analytically tractable models borrowed from statistical mechanics. We show that standard AGS is not more efficient than classical Metropolis-Hastings (MH) sampling of the effective energy landscape defined on the data layer. However, RBM can identify meaningful representations of training data in their latent space. Furthermore, using these representations and combining Gibbs sampling with the MH algorithm in the latent space can enhance the sampling performance of the RBM when the hidden units encode weakly dependent features of the data. We illustrate our findings on three datasets: Bars and Stripes and MNIST, well known in machine learning, and the so-called Lattice Proteins, introduced in theoretical biology to study the sequence-to-structure mapping in proteins.
Subjects: Disordered Systems and Neural Networks (cond-mat.dis-nn); Statistical Mechanics (cond-mat.stat-mech); Machine Learning (cs.LG)
Journal reference: Physical Review E : Statistical, Nonlinear, and Soft Matter Physics, American Physical Society, 2021
DOI: 10.1103/PhysRevE.104.034109
Cite as: arXiv:2107.06013 [cond-mat.dis-nn]
  (or arXiv:2107.06013v2 [cond-mat.dis-nn] for this version)

Submission history

From: Clement Roussel [view email]
[v1] Tue, 13 Jul 2021 12:07:56 GMT (1436kb,D)
[v2] Thu, 21 Oct 2021 07:39:00 GMT (1422kb,D)

Link back to: arXiv, form interface, contact.