References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Augmenting Knowledge Distillation With Peer-To-Peer Mutual Learning For Model Compression
(Submitted on 21 Oct 2021 (v1), last revised 22 Oct 2021 (this version, v2))
Abstract: Knowledge distillation (KD) is an effective model compression technique where a compact student network is taught to mimic the behavior of a complex and highly trained teacher network. In contrast, Mutual Learning (ML) provides an alternative strategy where multiple simple student networks benefit from sharing knowledge, even in the absence of a powerful but static teacher network. Motivated by these findings, we propose a single-teacher, multi-student framework that leverages both KD and ML to achieve better performance. Furthermore, an online distillation strategy is utilized to train the teacher and students simultaneously. To evaluate the performance of the proposed approach, extensive experiments were conducted using three different versions of teacher-student networks on benchmark biomedical classification (MSI vs. MSS) and object detection (Polyp Detection) tasks. Ensemble of student networks trained in the proposed manner achieved better results than the ensemble of students trained using KD or ML individually, establishing the benefit of augmenting knowledge transfer from teacher to students with peer-to-peer learning between students.
Submission history
From: Usma Bhat Niyaz [view email][v1] Thu, 21 Oct 2021 09:59:31 GMT (8153kb,D)
[v2] Fri, 22 Oct 2021 08:15:21 GMT (1520kb,D)
Link back to: arXiv, form interface, contact.