Current browse context:
stat.ML
Change to browse by:
References & Citations
Statistics > Machine Learning
Title: Loss function based second-order Jensen inequality and its application to particle variational inference
(Submitted on 9 Jun 2021 (v1), last revised 10 Jun 2021 (this version, v2))
Abstract: Bayesian model averaging, obtained as the expectation of a likelihood function by a posterior distribution, has been widely used for prediction, evaluation of uncertainty, and model selection. Various approaches have been developed to efficiently capture the information in the posterior distribution; one such approach is the optimization of a set of models simultaneously with interaction to ensure the diversity of the individual models in the same way as ensemble learning. A representative approach is particle variational inference (PVI), which uses an ensemble of models as an empirical approximation for the posterior distribution. PVI iteratively updates each model with a repulsion force to ensure the diversity of the optimized models. However, despite its promising performance, a theoretical understanding of this repulsion and its association with the generalization ability remains unclear. In this paper, we tackle this problem in light of PAC-Bayesian analysis. First, we provide a new second-order Jensen inequality, which has the repulsion term based on the loss function. Thanks to the repulsion term, it is tighter than the standard Jensen inequality. Then, we derive a novel generalization error bound and show that it can be reduced by enhancing the diversity of models. Finally, we derive a new PVI that optimizes the generalization error bound directly. Numerical experiments demonstrate that the performance of the proposed PVI compares favorably with existing methods in the experiment.
Submission history
From: Futoshi Futami [view email][v1] Wed, 9 Jun 2021 12:13:51 GMT (4375kb)
[v2] Thu, 10 Jun 2021 00:43:30 GMT (3728kb)
Link back to: arXiv, form interface, contact.