We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat.ML

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Machine Learning

Title: Variance-Reduced Proximal Stochastic Gradient Descent for Non-convex Composite optimization

Abstract: Here we study non-convex composite optimization: first, a finite-sum of smooth but non-convex functions, and second, a general function that admits a simple proximal mapping. Most research on stochastic methods for composite optimization assumes convexity or strong convexity of each function. In this paper, we extend this problem into the non-convex setting using variance reduction techniques, such as prox-SVRG and prox-SAGA. We prove that, with a constant step size, both prox-SVRG and prox-SAGA are suitable for non-convex composite optimization, and help the problem converge to a stationary point within $O(1/\epsilon)$ iterations. That is similar to the convergence rate seen with the state-of-the-art RSAG method and faster than stochastic gradient descent. Our analysis is also extended into the min-batch setting, which linearly accelerates the convergence. To the best of our knowledge, this is the first analysis of convergence rate of variance-reduced proximal stochastic gradient for non-convex composite optimization.
Comments: This paper has been withdrawn by the author due to an error in the proof of the convergence rate. They will modify this proof as soon as possible
Subjects: Machine Learning (stat.ML); Machine Learning (cs.LG); Numerical Analysis (math.NA)
Cite as: arXiv:1606.00602 [stat.ML]
  (or arXiv:1606.00602v2 [stat.ML] for this version)

Submission history

From: Xiyu Yu PhD [view email]
[v1] Thu, 2 Jun 2016 09:59:16 GMT (246kb,D)
[v2] Sun, 11 Sep 2016 04:15:04 GMT (0kb,I)

Link back to: arXiv, form interface, contact.