Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Self-Distillation as Instance-Specific Label Smoothing
(Submitted on 9 Jun 2020 (v1), last revised 22 Oct 2020 (this version, v2))
Abstract: It has been recently demonstrated that multi-generational self-distillation can improve generalization. Despite this intriguing observation, reasons for the enhancement remain poorly understood. In this paper, we first demonstrate experimentally that the improved performance of multi-generational self-distillation is in part associated with the increasing diversity in teacher predictions. With this in mind, we offer a new interpretation for teacher-student training as amortized MAP estimation, such that teacher predictions enable instance-specific regularization. Our framework allows us to theoretically relate self-distillation to label smoothing, a commonly used technique that regularizes predictive uncertainty, and suggests the importance of predictive diversity in addition to predictive uncertainty. We present experimental results using multiple datasets and neural network architectures that, overall, demonstrate the utility of predictive diversity. Finally, we propose a novel instance-specific label smoothing technique that promotes predictive diversity without the need for a separately trained teacher model. We provide an empirical evaluation of the proposed method, which, we find, often outperforms classical label smoothing.
Submission history
From: Zhilu Zhang [view email][v1] Tue, 9 Jun 2020 06:06:30 GMT (135kb,D)
[v2] Thu, 22 Oct 2020 03:23:39 GMT (163kb,D)
Link back to: arXiv, form interface, contact.