Current browse context:
stat.ML
Change to browse by:
References & Citations
Statistics > Machine Learning
Title: Margin Preservation of Deep Neural Networks
(Submitted on 26 May 2016 (this version), latest version 23 May 2017 (v3))
Abstract: The generalization error of deep neural networks via their classification margin is studied in this work, providing novel generalization error bounds that are independent of the network depth, thereby avoiding the common exponential depth-dependency which is unrealistic for current networks with hundreds of layers. We show that a large margin linear classifier operating at the output of a deep neural network induces a large classification margin at the input of the network, provided that the network preserves distances in directions normal to the decision boundary. The distance preservation is characterized by the average behaviour of the network's Jacobian matrix in the neighbourhood of the training samples. The introduced theory also leads to a margin preservation regularization scheme that outperforms weight decay both theoretically and empirically.
Submission history
From: Jure Sokolic [view email][v1] Thu, 26 May 2016 12:19:09 GMT (1168kb,D)
[v2] Mon, 3 Oct 2016 15:54:33 GMT (806kb,D)
[v3] Tue, 23 May 2017 11:45:31 GMT (912kb,D)
Link back to: arXiv, form interface, contact.