We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat.ML

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Machine Learning

Title: Gradual Learning of Recurrent Neural Networks

Abstract: Recurrent Neural Networks (RNNs) achieve state-of-the-art results in many sequence-to-sequence modeling tasks. However, RNNs are difficult to train and tend to suffer from overfitting. Motivated by the Data Processing Inequality (DPI), we formulate the multi-layered network as a Markov chain, introducing a training method that comprises training the network gradually and using layer-wise gradient clipping. We found that applying our methods, combined with previously introduced regularization and optimization methods, resulted in improvements in state-of-the-art architectures operating in language modeling tasks.
Comments: 8 pages, 2 figures
Subjects: Machine Learning (stat.ML); Information Theory (cs.IT); Machine Learning (cs.LG)
Cite as: arXiv:1708.08863 [stat.ML]
  (or arXiv:1708.08863v2 [stat.ML] for this version)

Submission history

From: Ziv Aharoni [view email]
[v1] Tue, 29 Aug 2017 16:18:44 GMT (509kb)
[v2] Mon, 21 May 2018 10:22:10 GMT (266kb)

Link back to: arXiv, form interface, contact.