We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

math.OC

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Mathematics > Optimization and Control

Title: Stochastic Gradient Methods with Preconditioned Updates

Abstract: This work considers non-convex finite sum minimization. There are a number of algorithms for such problems, but existing methods often work poorly when the problem is badly scaled and/or ill-conditioned, and a primary goal of this work is to introduce methods that alleviate this issue. Thus, here we include a preconditioner that is based upon Hutchinson's approach to approximating the diagonal of the Hessian, and couple it with several gradient based methods to give new `scaled' algorithms: {\tt Scaled SARAH} and {\tt Scaled L-SVRG}. Theoretical complexity guarantees under smoothness assumptions are presented, and we prove linear convergence when both smoothness and the PL-condition is assumed. Because our adaptively scaled methods use approximate partial second order curvature information, they are better able to mitigate the impact of badly scaled problems, and this improved practical performance is demonstrated in the numerical experiments that are also presented in this work.
Comments: 31 pages, 2 new algorithms, 19 figures, 2 tables
Subjects: Optimization and Control (math.OC); Machine Learning (cs.LG)
Cite as: arXiv:2206.00285 [math.OC]
  (or arXiv:2206.00285v1 [math.OC] for this version)

Submission history

From: Aleksandr Beznosikov [view email]
[v1] Wed, 1 Jun 2022 07:38:08 GMT (38062kb,D)

Link back to: arXiv, form interface, contact.