We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Gradient Normalization & Depth Based Decay For Deep Learning

Abstract: In this paper we introduce a novel method of gradient normalization and decay with respect to depth. Our method leverages the simple concept of normalizing all gradients in a deep neural network, and then decaying said gradients with respect to their depth in the network. Our proposed normalization and decay techniques can be used in conjunction with most current state of the art optimizers and are a very simple addition to any network. This method, although simple, showed improvements in convergence time on state of the art networks such as DenseNet and ResNet on image classification tasks, as well as on an LSTM for natural language processing tasks.
Comments: Results seemed more promising at the time
Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
Cite as: arXiv:1712.03607 [cs.LG]
  (or arXiv:1712.03607v2 [cs.LG] for this version)

Submission history

From: Robert Kwiatkowski [view email]
[v1] Sun, 10 Dec 2017 23:01:13 GMT (223kb,D)
[v2] Wed, 28 Feb 2018 15:56:52 GMT (0kb,I)

Link back to: arXiv, form interface, contact.