Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: On the Trend-corrected Variant of Adaptive Stochastic Optimization Methods
(Submitted on 17 Jan 2020 (v1), last revised 16 Dec 2020 (this version, v2))
Abstract: Adam-type optimizers, as a class of adaptive moment estimation methods with the exponential moving average scheme, have been successfully used in many applications of deep learning. Such methods are appealing due to the capability on large-scale sparse datasets with high computational efficiency. In this paper, we present a new framework for Adam-type methods with the trend information when updating the parameters with the adaptive step size and gradients. The additional terms in the algorithm promise an efficient movement on the complex cost surface, and thus the loss would converge more rapidly. We show empirically the importance of adding the trend component, where our framework outperforms the conventional Adam and AMSGrad methods constantly on the classical models with several real-world datasets.
Submission history
From: Xuebin Zheng [view email][v1] Fri, 17 Jan 2020 01:23:23 GMT (119kb)
[v2] Wed, 16 Dec 2020 01:39:55 GMT (167kb)
Link back to: arXiv, form interface, contact.