References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: ResKD: Residual-Guided Knowledge Distillation
(Submitted on 8 Jun 2020 (v1), last revised 9 Mar 2021 (this version, v4))
Abstract: Knowledge distillation, aimed at transferring the knowledge from a heavy teacher network to a lightweight student network, has emerged as a promising technique for compressing neural networks. However, due to the capacity gap between the heavy teacher and the lightweight student, there still exists a significant performance gap between them. In this paper, we see knowledge distillation in a fresh light, using the knowledge gap, or the residual, between a teacher and a student as guidance to train a much more lightweight student, called a res-student. We combine the student and the res-student into a new student, where the res-student rectifies the errors of the former student. Such a residual-guided process can be repeated until the user strikes the balance between accuracy and cost. At inference time, we propose a sample-adaptive strategy to decide which res-students are not necessary for each sample, which can save computational cost. Experimental results show that we achieve competitive performance with 18.04$\%$, 23.14$\%$, 53.59$\%$, and 56.86$\%$ of the teachers' computational costs on the CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet datasets. Finally, we do thorough theoretical and empirical analysis for our method.
Submission history
From: Xi Li [view email][v1] Mon, 8 Jun 2020 16:18:45 GMT (746kb,D)
[v2] Tue, 9 Jun 2020 11:39:18 GMT (579kb,D)
[v3] Sat, 1 Aug 2020 17:21:09 GMT (3886kb,D)
[v4] Tue, 9 Mar 2021 03:35:20 GMT (12784kb,D)
Link back to: arXiv, form interface, contact.