We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cond-mat.stat-mech

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Condensed Matter > Statistical Mechanics

Title: Annealing and replica-symmetry in Deep Boltzmann Machines

Abstract: In this paper we study the properties of the quenched pressure of a multi-layer spin-glass model (a deep Boltzmann Machine in artificial intelligence jargon) whose pairwise interactions are allowed between spins lying in adjacent layers and not inside the same layer nor among layers at distance larger than one. We prove a theorem that bounds the quenched pressure of such a K-layer machine in terms of K Sherrington-Kirkpatrick spin glasses and use it to investigate its annealed region. The replica-symmetric approximation of the quenched pressure is identified and its relation to the annealed one is considered. The paper also presents some observation on the model's architectural structure related to machine learning. Since escaping the annealed region is mandatory for a meaningful training, by squeezing such region we obtain thermodynamical constraints on the form factors. Remarkably, its optimal escape is achieved by requiring the last layer to scale sub-linearly in the network size.
Comments: 15 pages, 1 figure. To appear in the Special Issue dedicated to celebrating Joel Lebowitz
Subjects: Statistical Mechanics (cond-mat.stat-mech); Disordered Systems and Neural Networks (cond-mat.dis-nn); Mathematical Physics (math-ph)
MSC classes: 82-XX, 82D30,
DOI: 10.1007/s10955-020-02495-2
Cite as: arXiv:2001.07714 [cond-mat.stat-mech]
  (or arXiv:2001.07714v1 [cond-mat.stat-mech] for this version)

Submission history

From: Pierluigi Contucci [view email]
[v1] Tue, 21 Jan 2020 15:01:14 GMT (211kb,D)

Link back to: arXiv, form interface, contact.