We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Generalized AdaGrad (G-AdaGrad) and Adam: A State-Space Perspective

Abstract: Accelerated gradient-based methods are being extensively used for solving non-convex machine learning problems, especially when the data points are abundant or the available data is distributed across several agents. Two of the prominent accelerated gradient algorithms are AdaGrad and Adam. AdaGrad is the simplest accelerated gradient method, which is particularly effective for sparse data. Adam has been shown to perform favorably in deep learning problems compared to other methods. In this paper, we propose a new fast optimizer, Generalized AdaGrad (G-AdaGrad), for accelerating the solution of potentially non-convex machine learning problems. Specifically, we adopt a state-space perspective for analyzing the convergence of gradient acceleration algorithms, namely G-AdaGrad and Adam, in machine learning. Our proposed state-space models are governed by ordinary differential equations. We present simple convergence proofs of these two algorithms in the deterministic settings with minimal assumptions. Our analysis also provides intuition behind improving upon AdaGrad's convergence rate. We provide empirical results on MNIST dataset to reinforce our claims on the convergence and performance of G-AdaGrad and Adam.
Comments: Updates: The parameter condition of Adam in Theorem 2 has been relaxed and the proof has been updated accordingly. Experimental results on logistic regression model have been included. Conference: Accepted for presentation in the 2021 60th IEEE Conference on Decision and Control (CDC)
Subjects: Machine Learning (cs.LG); Systems and Control (eess.SY); Machine Learning (stat.ML)
Cite as: arXiv:2106.00092 [cs.LG]
  (or arXiv:2106.00092v2 [cs.LG] for this version)

Submission history

From: Kushal Chakrabarti [view email]
[v1] Mon, 31 May 2021 20:30:25 GMT (581kb,D)
[v2] Thu, 30 Sep 2021 22:45:26 GMT (817kb,D)

Link back to: arXiv, form interface, contact.