Current browse context:
math
Change to browse by:
References & Citations
Mathematics > Statistics Theory
Title: Asymptotic Analysis of Conditioned Stochastic Gradient Descent
(Submitted on 4 Jun 2020 (v1), last revised 15 Oct 2023 (this version, v5))
Abstract: In this paper, we investigate a general class of stochastic gradient descent (SGD) algorithms, called Conditioned SGD, based on a preconditioning of the gradient direction. Using a discrete-time approach with martingale tools, we establish under mild assumptions the weak convergence of the rescaled sequence of iterates for a broad class of conditioning matrices including stochastic first-order and second-order methods. Almost sure convergence results, which may be of independent interest, are also presented. Interestingly, the asymptotic normality result consists in a stochastic equicontinuity property so when the conditioning matrix is an estimate of the inverse Hessian, the algorithm is asymptotically optimal.
Submission history
From: Remi Leluc [view email][v1] Thu, 4 Jun 2020 10:08:05 GMT (364kb,D)
[v2] Thu, 1 Oct 2020 08:53:50 GMT (360kb,D)
[v3] Fri, 9 Jul 2021 14:43:08 GMT (364kb,D)
[v4] Thu, 20 Oct 2022 14:36:38 GMT (724kb,D)
[v5] Sun, 15 Oct 2023 13:23:07 GMT (243kb,D)
Link back to: arXiv, form interface, contact.