We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Exact marginal prior distributions of finite Bayesian neural networks

Abstract: Bayesian neural networks are theoretically well-understood only in the infinite-width limit, where Gaussian priors over network weights yield Gaussian priors over network outputs. Recent work has suggested that finite Bayesian networks may outperform their infinite counterparts, but their non-Gaussian function space priors have been characterized only though perturbative approaches. Here, we derive exact solutions for the function space priors for individual input examples of a class of finite fully-connected feedforward Bayesian neural networks. For deep linear networks, the prior has a simple expression in terms of the Meijer $G$-function. The prior of a finite ReLU network is a mixture of the priors of linear networks of smaller widths, corresponding to different numbers of active units in each layer. Our results unify previous descriptions of finite network priors in terms of their tail decay and large-width behavior.
Comments: 12+9 pages, 4 figures; v3: Accepted as NeurIPS 2021 Spotlight
Subjects: Machine Learning (cs.LG); Disordered Systems and Neural Networks (cond-mat.dis-nn); Machine Learning (stat.ML)
Journal reference: Advances in Neural Information Processing Systems 34 (2021)
Cite as: arXiv:2104.11734 [cs.LG]
  (or arXiv:2104.11734v3 [cs.LG] for this version)

Submission history

From: Jacob Zavatone-Veth [view email]
[v1] Fri, 23 Apr 2021 17:31:42 GMT (1664kb,D)
[v2] Tue, 18 May 2021 17:42:44 GMT (2093kb,D)
[v3] Mon, 18 Oct 2021 13:59:44 GMT (2187kb,D)

Link back to: arXiv, form interface, contact.