Current browse context:
stat.ML
Change to browse by:
References & Citations
Statistics > Machine Learning
Title: Last Layer Marginal Likelihood for Invariance Learning
(Submitted on 14 Jun 2021 (v1), last revised 1 Mar 2022 (this version, v2))
Abstract: Data augmentation is often used to incorporate inductive biases into models. Traditionally, these are hand-crafted and tuned with cross validation. The Bayesian paradigm for model selection provides a path towards end-to-end learning of invariances using only the training data, by optimising the marginal likelihood. Computing the marginal likelihood is hard for neural networks, but success with tractable approaches that compute the marginal likelihood for the last layer only raises the question of whether this convenient approach might be employed for learning invariances. We show partial success on standard benchmarks, in the low-data regime and on a medical imaging dataset by designing a custom optimisation routine. Introducing a new lower bound to the marginal likelihood allows us to perform inference for a larger class of likelihood functions than before. On the other hand, we demonstrate failure modes on the CIFAR10 dataset, where the last layer approximation is not sufficient due to the increased complexity of our neural network. Our results indicate that once more sophisticated approximations become available the marginal likelihood is a promising approach for invariance learning in neural networks.
Submission history
From: Pola Schwöbel [view email][v1] Mon, 14 Jun 2021 15:40:51 GMT (1131kb,D)
[v2] Tue, 1 Mar 2022 13:27:06 GMT (1134kb,D)
Link back to: arXiv, form interface, contact.