We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Dangers of Bayesian Model Averaging under Covariate Shift

Abstract: Approximate Bayesian inference for neural networks is considered a robust alternative to standard training, often providing good performance on out-of-distribution data. However, Bayesian neural networks (BNNs) with high-fidelity approximate inference via full-batch Hamiltonian Monte Carlo achieve poor generalization under covariate shift, even underperforming classical estimation. We explain this surprising result, showing how a Bayesian model average can in fact be problematic under covariate shift, particularly in cases where linear dependencies in the input features cause a lack of posterior contraction. We additionally show why the same issue does not affect many approximate inference procedures, or classical maximum a-posteriori (MAP) training. Finally, we propose novel priors that improve the robustness of BNNs to many sources of covariate shift.
Comments: NeurIPS 2021. Code is available at this https URL
Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
Cite as: arXiv:2106.11905 [cs.LG]
  (or arXiv:2106.11905v2 [cs.LG] for this version)

Submission history

From: Andrew Wilson [view email]
[v1] Tue, 22 Jun 2021 16:19:52 GMT (2524kb,D)
[v2] Mon, 6 Dec 2021 14:45:30 GMT (2558kb,D)

Link back to: arXiv, form interface, contact.