Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Statistical Mechanical Analysis of Neural Network Pruning
(Submitted on 30 Jun 2020 (v1), last revised 11 Jun 2021 (this version, v3))
Abstract: Deep learning architectures with a huge number of parameters are often compressed using pruning techniques to ensure computational efficiency of inference during deployment. Despite multitude of empirical advances, there is a lack of theoretical understanding of the effectiveness of different pruning methods. We inspect different pruning techniques under the statistical mechanics formulation of a teacher-student framework and derive their generalization error (GE) bounds. It has been shown that Determinantal Point Process (DPP) based node pruning method is notably superior to competing approaches when tested on real datasets. Using GE bounds in the aforementioned setup we provide theoretical guarantees for their empirical observations. Another consistent finding in literature is that sparse neural networks (edge pruned) generalize better than dense neural networks (node pruned) for a fixed number of parameters. We use our theoretical setup to prove this finding and show that even the baseline random edge pruning method performs better than the DPP node pruning method. We also validate this empirically on real datasets.
Submission history
From: Rupam Acharyya [view email][v1] Tue, 30 Jun 2020 09:15:25 GMT (5355kb,D)
[v2] Thu, 25 Feb 2021 03:34:28 GMT (5355kb,D)
[v3] Fri, 11 Jun 2021 04:34:46 GMT (9538kb,D)
Link back to: arXiv, form interface, contact.