Current browse context:
stat.ML
Change to browse by:
References & Citations
Statistics > Machine Learning
Title: Recurrent neural network training with preconditioned stochastic gradient descent
(Submitted on 14 Jun 2016 (v1), last revised 8 Dec 2016 (this version, v2))
Abstract: This paper studies the performance of a recently proposed preconditioned stochastic gradient descent (PSGD) algorithm on recurrent neural network (RNN) training. PSGD adaptively estimates a preconditioner to accelerate gradient descent, and is designed to be simple, general and easy to use, as stochastic gradient descent (SGD). RNNs, especially the ones requiring extremely long term memories, are difficult to train. We have tested PSGD on a set of synthetic pathological RNN learning problems and the real world MNIST handwritten digit recognition task. Experimental results suggest that PSGD is able to achieve highly competitive performance without using any trick like preprocessing, pretraining or parameter tweaking.
Submission history
From: Xi-Lin Li [view email][v1] Tue, 14 Jun 2016 16:40:38 GMT (1848kb,D)
[v2] Thu, 8 Dec 2016 18:56:38 GMT (383kb,D)
Link back to: arXiv, form interface, contact.