Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Winning the Lottery with Continuous Sparsification
(Submitted on 10 Dec 2019 (v1), last revised 11 Jan 2021 (this version, v4))
Abstract: The search for efficient, sparse deep neural network models is most prominently performed by pruning: training a dense, overparameterized network and removing parameters, usually via following a manually-crafted heuristic. Additionally, the recent Lottery Ticket Hypothesis conjectures that, for a typically-sized neural network, it is possible to find small sub-networks which, when trained from scratch on a comparable budget, match the performance of the original dense counterpart. We revisit fundamental aspects of pruning algorithms, pointing out missing ingredients in previous approaches, and develop a method, Continuous Sparsification, which searches for sparse networks based on a novel approximation of an intractable $\ell_0$ regularization. We compare against dominant heuristic-based methods on pruning as well as ticket search -- finding sparse subnetworks that can be successfully re-trained from an early iterate. Empirical results show that we surpass the state-of-the-art for both objectives, across models and datasets, including VGG trained on CIFAR-10 and ResNet-50 trained on ImageNet. In addition to setting a new standard for pruning, Continuous Sparsification also offers fast parallel ticket search, opening doors to new applications of the Lottery Ticket Hypothesis.
Submission history
From: Pedro Savarese [view email][v1] Tue, 10 Dec 2019 00:30:34 GMT (449kb,D)
[v2] Tue, 17 Mar 2020 00:55:36 GMT (288kb,D)
[v3] Fri, 26 Jun 2020 23:58:27 GMT (179kb,D)
[v4] Mon, 11 Jan 2021 11:53:22 GMT (532kb,D)
Link back to: arXiv, form interface, contact.