We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

math.ST

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Mathematics > Statistics Theory

Title: Asymptotic Analysis of Conditioned Stochastic Gradient Descent

Abstract: In this paper, we investigate a general class of stochastic gradient descent (SGD) algorithms, called conditioned SGD, based on a preconditioning of the gradient direction. Using a discrete-time approach with martingale tools, we establish the weak convergence of the rescaled sequence of iterates for a broad class of conditioning matrices including stochastic first-order and second-order methods. Almost sure convergence results, which may be of independent interest, are also presented. When the conditioning matrix is an estimate of the inverse Hessian, the algorithm is proved to be asymptotically optimal. For the sake of completeness, we provide a practical procedure to achieve this minimum variance.
Subjects: Statistics Theory (math.ST); Machine Learning (cs.LG)
Cite as: arXiv:2006.02745 [math.ST]
  (or arXiv:2006.02745v4 [math.ST] for this version)

Submission history

From: Remi Leluc [view email]
[v1] Thu, 4 Jun 2020 10:08:05 GMT (364kb,D)
[v2] Thu, 1 Oct 2020 08:53:50 GMT (360kb,D)
[v3] Fri, 9 Jul 2021 14:43:08 GMT (364kb,D)
[v4] Thu, 20 Oct 2022 14:36:38 GMT (724kb,D)

Link back to: arXiv, form interface, contact.