We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.NE

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Neural and Evolutionary Computing

Title: Structured Pruning of Neural Networks with Budget-Aware Regularization

Abstract: Pruning methods have shown to be effective at reducing the size of deep neural networks while keeping accuracy almost intact. Among the most effective methods are those that prune a network while training it with a sparsity prior loss and learnable dropout parameters. A shortcoming of these approaches however is that neither the size nor the inference speed of the pruned network can be controlled directly; yet this is a key feature for targeting deployment of CNNs on low-power hardware. To overcome this, we introduce a budgeted regularized pruning framework for deep CNNs. Our approach naturally fits into traditional neural network training as it consists of a learnable masking layer, a novel budget-aware objective function, and the use of knowledge distillation. We also provide insights on how to prune a residual network and how this can lead to new architectures. Experimental results reveal that CNNs pruned with our method are more accurate and less compute-hungry than state-of-the-art methods. Also, our approach is more effective at preventing accuracy collapse in case of severe pruning; this allows us to attain pruning factors up to 16x without significant accuracy drop.
Comments: Paper: 9 pages, 8 figures. Supplementary materials: 7 pages, 3 figures
Subjects: Neural and Evolutionary Computing (cs.NE)
Cite as: arXiv:1811.09332 [cs.NE]
  (or arXiv:1811.09332v3 [cs.NE] for this version)

Submission history

From: Carl Lemaire [view email]
[v1] Fri, 23 Nov 2018 00:30:40 GMT (2027kb)
[v2] Fri, 11 Oct 2019 22:47:57 GMT (2555kb)
[v3] Thu, 19 Dec 2019 14:52:06 GMT (2555kb)

Link back to: arXiv, form interface, contact.