Current browse context:
cs
Change to browse by:
References & Citations
Mathematics > Numerical Analysis
Title: Sketched Newton-Raphson
(Submitted on 22 Jun 2020 (v1), last revised 9 May 2022 (this version, v5))
Abstract: We propose a new globally convergent stochastic second order method. Our starting point is the development of a new Sketched Newton-Raphson (SNR) method for solving large scale nonlinear equations of the form $F(x)=0$ with $F:\mathbb{R}^p \rightarrow \mathbb{R}^m$. We then show how to design several stochastic second order optimization methods by re-writing the optimization problem of interest as a system of nonlinear equations and applying SNR. For instance, by applying SNR to find a stationary point of a generalized linear model (GLM), we derive completely new and scalable stochastic second order methods. We show that the resulting method is very competitive as compared to state-of-the-art variance reduced methods. Furthermore, using a variable splitting trick, we also show that the Stochastic Newton method (SNM) is a special case of SNR, and use this connection to establish the first global convergence theory of SNM.
We establish the global convergence of SNR by showing that it is a variant of the stochastic gradient descent (SGD) method, and then leveraging proof techniques of SGD. As a special case, our theory also provides a new global convergence theory for the original Newton-Raphson method under strictly weaker assumptions as compared to the classic monotone convergence theory.
Submission history
From: Rui Yuan [view email][v1] Mon, 22 Jun 2020 10:21:49 GMT (477kb,D)
[v2] Thu, 9 Jul 2020 08:49:46 GMT (477kb,D)
[v3] Sun, 17 Jan 2021 21:24:44 GMT (475kb,D)
[v4] Mon, 3 Jan 2022 16:47:53 GMT (515kb,D)
[v5] Mon, 9 May 2022 12:43:26 GMT (518kb,D)
Link back to: arXiv, form interface, contact.