Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Improved SVRG for quadratic functions
(Submitted on 1 Jun 2020 (v1), last revised 15 Jun 2021 (this version, v2))
Abstract: We analyse an iterative algorithm to minimize quadratic functions whose Hessian matrix $H$ is the expectation of a random symmetric $d\times d$ matrix. The algorithm is a variant of the stochastic variance reduced gradient (SVRG). In several applications, including least-squares regressions, ridge regressions, linear discriminant analysis and regularized linear discriminant analysis, the running time of each iteration is proportional to $d$. Under smoothness and convexity conditions, the algorithm has linear convergence. When applied to quadratic functions, our analysis improves the state-of-the-art performance of SVRG up to a logarithmic factor. Furthermore, for well-conditioned quadratic problems, our analysis improves the state-of-the-art running times of accelerated SVRG, and is better than the known matching lower bound, by a logarithmic factor. Our theoretical results are backed with numerical experiments.
Submission history
From: Nabil Kahale [view email][v1] Mon, 1 Jun 2020 15:28:08 GMT (18kb)
[v2] Tue, 15 Jun 2021 07:36:41 GMT (20kb)
Link back to: arXiv, form interface, contact.