Current browse context:
stat
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Go Wide, Then Narrow: Efficient Training of Deep Thin Networks
(Submitted on 1 Jul 2020 (v1), last revised 17 Aug 2020 (this version, v2))
Abstract: For deploying a deep learning model into production, it needs to be both accurate and compact to meet the latency and memory constraints. This usually results in a network that is deep (to ensure performance) and yet thin (to improve computational efficiency). In this paper, we propose an efficient method to train a deep thin network with a theoretic guarantee. Our method is motivated by model compression. It consists of three stages. First, we sufficiently widen the deep thin network and train it until convergence. Then, we use this well-trained deep wide network to warm up (or initialize) the original deep thin network. This is achieved by layerwise imitation, that is, forcing the thin network to mimic the intermediate outputs of the wide network from layer to layer. Finally, we further fine tune this already well-initialized deep thin network. The theoretical guarantee is established by using the neural mean field analysis. It demonstrates the advantage of our layerwise imitation approach over backpropagation. We also conduct large-scale empirical experiments to validate the proposed method. By training with our method, ResNet50 can outperform ResNet101, and BERT Base can be comparable with BERT Large, when ResNet101 and BERT Large are trained under the standard training procedures as in the literature.
Submission history
From: Denny Zhou [view email][v1] Wed, 1 Jul 2020 23:34:35 GMT (135kb,D)
[v2] Mon, 17 Aug 2020 17:43:30 GMT (154kb,D)
Link back to: arXiv, form interface, contact.