References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Lottery Jackpots Exist in Pre-trained Models
(Submitted on 18 Apr 2021 (v1), revised 2 Jun 2021 (this version, v2), latest version 2 Sep 2023 (v7))
Abstract: Network pruning is an effective approach to reduce network complexity without performance compromise. Existing studies achieve the sparsity of neural networks via time-consuming weight tuning or complex search on networks with expanded width, which greatly limits the applications of network pruning. In this paper, we show that high-performing and sparse sub-networks without the involvement of weight tuning, termed "lottery jackpots", exist in pre-trained models with unexpanded width. For example, we obtain a lottery jackpot that has only 10% parameters and still reaches the performance of the original dense VGGNet-19 without any modifications on the pre-trained weights. Furthermore, we observe that the sparse masks derived from many existing pruning criteria have a high overlap with the searched mask of our lottery jackpot, among which, the magnitude-based pruning results in the most similar mask with ours. Based on this insight, we initialize our sparse mask using the magnitude pruning, resulting in at least 3x cost reduction on the lottery jackpot search while achieves comparable or even better performance. Specifically, our magnitude-based lottery jackpot removes 90% weights in the ResNet-50, while easily obtains more than 70% top-1 accuracy using only 10 searching epochs on ImageNet.
Submission history
From: Yuxin Zhang [view email][v1] Sun, 18 Apr 2021 03:50:28 GMT (544kb,D)
[v2] Wed, 2 Jun 2021 06:21:53 GMT (111kb,D)
[v3] Thu, 9 Sep 2021 13:14:28 GMT (96kb,D)
[v4] Mon, 22 Nov 2021 03:05:24 GMT (901kb,D)
[v5] Fri, 4 Feb 2022 14:10:00 GMT (2686kb,D)
[v6] Tue, 13 Dec 2022 03:08:41 GMT (3025kb,D)
[v7] Sat, 2 Sep 2023 05:09:41 GMT (1489kb,D)
Link back to: arXiv, form interface, contact.