We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Model Preserving Compression for Neural Networks

Abstract: After training complex deep learning models, a common task is to compress the model to reduce compute and storage demands. When compressing, it is desirable to preserve the original model's per-example decisions (e.g., to go beyond top-1 accuracy or preserve robustness), maintain the network's structure, automatically determine per-layer compression levels, and eliminate the need for fine tuning. No existing compression methods simultaneously satisfy these criteria $\unicode{x2014}$ we introduce a principled approach that does by leveraging interpolative decompositions. Our approach simultaneously selects and eliminates channels (analogously, neurons), then constructs an interpolation matrix that propagates a correction into the next layer, preserving the network's structure. Consequently, our method achieves good performance even without fine tuning and admits theoretical analysis. Our theoretical generalization bound for a one layer network lends itself naturally to a heuristic that allows our method to automatically choose per-layer sizes for deep networks. We demonstrate the efficacy of our approach with strong empirical performance on a variety of tasks, models, and datasets $\unicode{x2014}$ from simple one-hidden-layer networks to deep networks on ImageNet.
Comments: 26 pages, 15 figures. To be published in Advances in Neural Information Processing Systems 35
Subjects: Machine Learning (cs.LG)
MSC classes: 68W99, 65F55
Cite as: arXiv:2108.00065 [cs.LG]
  (or arXiv:2108.00065v2 [cs.LG] for this version)

Submission history

From: Megan Renz [view email]
[v1] Fri, 30 Jul 2021 20:13:49 GMT (910kb,D)
[v2] Fri, 14 Oct 2022 18:56:47 GMT (2005kb,D)

Link back to: arXiv, form interface, contact.