References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Lottery Jackpots Exist in Pre-trained Models
(Submitted on 18 Apr 2021 (v1), last revised 2 Sep 2023 (this version, v7))
Abstract: Network pruning is an effective approach to reduce network complexity with acceptable performance compromise. Existing studies achieve the sparsity of neural networks via time-consuming weight training or complex searching on networks with expanded width, which greatly limits the applications of network pruning. In this paper, we show that high-performing and sparse sub-networks without the involvement of weight training, termed "lottery jackpots", exist in pre-trained models with unexpanded width. Furthermore, we improve the efficiency for searching lottery jackpots from two perspectives. Firstly, we observe that the sparse masks derived from many existing pruning criteria have a high overlap with the searched mask of our lottery jackpot, among which, the magnitude-based pruning results in the most similar mask with ours. Consequently, our searched lottery jackpot removes 90% weights in ResNet-50, while it easily obtains more than 70% top-1 accuracy using only 5 searching epochs on ImageNet. In compliance with this insight, we initialize our sparse mask using the magnitude-based pruning, resulting in at least 3x cost reduction on the lottery jackpot searching while achieving comparable or even better performance. Secondly, we conduct an in-depth analysis of the searching process for lottery jackpots. Our theoretical result suggests that the decrease in training loss during weight searching can be disturbed by the dependency between weights in modern networks. To mitigate this, we propose a novel short restriction method to restrict change of masks that may have potential negative impacts on the training loss. Our code is available at this https URL
Submission history
From: Yuxin Zhang [view email][v1] Sun, 18 Apr 2021 03:50:28 GMT (544kb,D)
[v2] Wed, 2 Jun 2021 06:21:53 GMT (111kb,D)
[v3] Thu, 9 Sep 2021 13:14:28 GMT (96kb,D)
[v4] Mon, 22 Nov 2021 03:05:24 GMT (901kb,D)
[v5] Fri, 4 Feb 2022 14:10:00 GMT (2686kb,D)
[v6] Tue, 13 Dec 2022 03:08:41 GMT (3025kb,D)
[v7] Sat, 2 Sep 2023 05:09:41 GMT (1489kb,D)
Link back to: arXiv, form interface, contact.