Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Bayesian Deep Learning via Subnetwork Inference
(Submitted on 28 Oct 2020 (v1), last revised 14 Mar 2022 (this version, v4))
Abstract: The Bayesian paradigm has the potential to solve core issues of deep neural networks such as poor calibration and data inefficiency. Alas, scaling Bayesian inference to large weight spaces often requires restrictive approximations. In this work, we show that it suffices to perform inference over a small subset of model weights in order to obtain accurate predictive posteriors. The other weights are kept as point estimates. This subnetwork inference framework enables us to use expressive, otherwise intractable, posterior approximations over such subsets. In particular, we implement subnetwork linearized Laplace as a simple, scalable Bayesian deep learning method: We first obtain a MAP estimate of all weights and then infer a full-covariance Gaussian posterior over a subnetwork using the linearized Laplace approximation. We propose a subnetwork selection strategy that aims to maximally preserve the model's predictive uncertainty. Empirically, our approach compares favorably to ensembles and less expressive posterior approximations over full networks. Our proposed subnetwork (linearized) Laplace method is implemented within the laplace PyTorch library at this https URL
Submission history
From: Erik Daxberger [view email][v1] Wed, 28 Oct 2020 01:10:11 GMT (1050kb,D)
[v2] Thu, 18 Feb 2021 09:20:51 GMT (1562kb,D)
[v3] Tue, 9 Nov 2021 14:30:59 GMT (1583kb,D)
[v4] Mon, 14 Mar 2022 13:46:12 GMT (1606kb,D)
Link back to: arXiv, form interface, contact.