We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Stochastic Function Norm Regularization of Deep Networks

Abstract: Deep neural networks have had an enormous impact on image analysis. State-of-the-art training methods, based on weight decay and DropOut, result in impressive performance when a very large training set is available. However, they tend to have large problems overfitting to small data sets. Indeed, the available regularization methods deal with the complexity of the network function only indirectly. In this paper, we study the feasibility of directly using the $L_2$ function norm for regularization. Two methods to integrate this new regularization in the stochastic backpropagation are proposed. Moreover, the convergence of these new algorithms is studied. We finally show that they outperform the state-of-the-art methods in the low sample regime on benchmark datasets (MNIST and CIFAR10). The obtained results demonstrate very clear improvement, especially in the context of small sample regimes with data laying in a low dimensional manifold. Source code of the method can be found at \url{this https URL}.
Comments: arXiv admin note: text overlap with arXiv:1710.06703
Subjects: Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (stat.ML)
Cite as: arXiv:1605.09085 [cs.LG]
  (or arXiv:1605.09085v3 [cs.LG] for this version)

Submission history

From: Matthew Blaschko [view email]
[v1] Mon, 30 May 2016 01:49:18 GMT (178kb,D)
[v2] Wed, 7 Dec 2016 14:14:30 GMT (189kb,D)
[v3] Fri, 30 Aug 2019 14:38:32 GMT (292kb,D)

Link back to: arXiv, form interface, contact.