We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Mathematics > Optimization and Control

Title: Convergence of Random Reshuffling Under The Kurdyka-Łojasiewicz Inequality

Abstract: We study the random reshuffling (RR) method for smooth nonconvex optimization problems with a finite-sum structure. Though this method is widely utilized in practice such as the training of neural networks, its convergence behavior is only understood in several limited settings. In this paper, under the well-known Kurdyka-Lojasiewicz (KL) inequality, we establish strong limit-point convergence results for RR with appropriate diminishing step sizes, namely, the whole sequence of iterates generated by RR is convergent and converges to a single stationary point in an almost sure sense. In addition, we derive the corresponding rate of convergence, depending on the KL exponent and the suitably selected diminishing step sizes. When the KL exponent lies in $[0,\frac12]$, the convergence is at a rate of $\mathcal{O}(t^{-1})$ with $t$ counting the iteration number. When the KL exponent belongs to $(\frac12,1)$, our derived convergence rate is of the form $\mathcal{O}(t^{-q})$ with $q\in (0,1)$ depending on the KL exponent. The standard KL inequality-based convergence analysis framework only applies to algorithms with a certain descent property. We conduct a novel convergence analysis for the non-descent RR method with diminishing step sizes based on the KL inequality, which generalizes the standard KL framework. We summarize our main steps and core ideas in an informal analysis framework, which is of independent interest. As a direct application of this framework, we also establish similar strong limit-point convergence results for the reshuffled proximal point method.
Comments: Accepted for publication in SIAM Journal on Optimization
Subjects: Optimization and Control (math.OC); Machine Learning (cs.LG)
Cite as: arXiv:2110.04926 [math.OC]
  (or arXiv:2110.04926v4 [math.OC] for this version)

Submission history

From: Xiao Li [view email]
[v1] Sun, 10 Oct 2021 23:20:04 GMT (3473kb)
[v2] Mon, 3 Jan 2022 02:59:22 GMT (3464kb)
[v3] Mon, 3 Oct 2022 16:03:59 GMT (3460kb)
[v4] Wed, 25 Jan 2023 09:14:16 GMT (3477kb)

Link back to: arXiv, form interface, contact.