Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Normalized Direction-preserving Adam
(Submitted on 13 Sep 2017 (v1), last revised 18 Sep 2018 (this version, v2))
Abstract: Adaptive optimization algorithms, such as Adam and RMSprop, have shown better optimization performance than stochastic gradient descent (SGD) in some scenarios. However, recent studies show that they often lead to worse generalization performance than SGD, especially for training deep neural networks (DNNs). In this work, we identify the reasons that Adam generalizes worse than SGD, and develop a variant of Adam to eliminate the generalization gap. The proposed method, normalized direction-preserving Adam (ND-Adam), enables more precise control of the direction and step size for updating weight vectors, leading to significantly improved generalization performance. Following a similar rationale, we further improve the generalization performance in classification tasks by regularizing the softmax logits. By bridging the gap between SGD and Adam, we also hope to shed light on why certain optimization algorithms generalize better than others.
Submission history
From: Zijun Zhang [view email][v1] Wed, 13 Sep 2017 21:38:02 GMT (1399kb,D)
[v2] Tue, 18 Sep 2018 02:35:39 GMT (971kb,D)
Link back to: arXiv, form interface, contact.