Current browse context:
stat.ML
Change to browse by:
References & Citations
Statistics > Machine Learning
Title: A general framework for ensemble distribution distillation
(Submitted on 26 Feb 2020 (v1), last revised 8 Jan 2021 (this version, v2))
Abstract: Ensembles of neural networks have been shown to give better performance than single networks, both in terms of predictions and uncertainty estimation. Additionally, ensembles allow the uncertainty to be decomposed into aleatoric (data) and epistemic (model) components, giving a more complete picture of the predictive uncertainty. Ensemble distillation is the process of compressing an ensemble into a single model, often resulting in a leaner model that still outperforms the individual ensemble members. Unfortunately, standard distillation erases the natural uncertainty decomposition of the ensemble. We present a general framework for distilling both regression and classification ensembles in a way that preserves the decomposition. We demonstrate the desired behaviour of our framework and show that its predictive performance is on par with standard distillation.
Submission history
From: Jakob Lindqvist [view email][v1] Wed, 26 Feb 2020 14:34:43 GMT (135kb)
[v2] Fri, 8 Jan 2021 11:20:35 GMT (211kb)
Link back to: arXiv, form interface, contact.