We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat.ML

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Machine Learning

Title: Estimate Sequences for Variance-Reduced Stochastic Composite Optimization

Authors: Andrei Kulunchakov (Thoth), Julien Mairal (Thoth)
Abstract: In this paper, we propose a unified view of gradient-based algorithms for stochastic convex composite optimization by extending the concept of estimate sequence introduced by Nesterov. This point of view covers the stochastic gradient descent method, variants of the approaches SAGA, SVRG, and has several advantages: (i) we provide a generic proof of convergence for the aforementioned methods; (ii) we show that this SVRG variant is adaptive to strong convexity; (iii) we naturally obtain new algorithms with the same guarantees; (iv) we derive generic strategies to make these algorithms robust to stochastic noise, which is useful when data is corrupted by small random perturbations. Finally, we show that this viewpoint is useful to obtain new accelerated algorithms in the sense of Nesterov.
Comments: short version of preprint arXiv:1901.08788
Subjects: Machine Learning (stat.ML); Machine Learning (cs.LG); Optimization and Control (math.OC)
Journal reference: International Conference on Machine Learning (ICML), Jun 2019, Long Beach, United States
Cite as: arXiv:1905.02374 [stat.ML]
  (or arXiv:1905.02374v1 [stat.ML] for this version)

Submission history

From: Julien Mairal [view email]
[v1] Tue, 7 May 2019 06:41:24 GMT (131kb,D)

Link back to: arXiv, form interface, contact.