Current browse context:
stat.ML
Change to browse by:
References & Citations
Statistics > Machine Learning
Title: Catalyst Acceleration for First-order Convex Optimization: from Theory to Practice
(Submitted on 15 Dec 2017 (v1), last revised 19 Jun 2018 (this version, v2))
Abstract: We introduce a generic scheme for accelerating gradient-based optimization methods in the sense of Nesterov. The approach, called Catalyst, builds upon the inexact accelerated proximal point algorithm for minimizing a convex objective function, and consists of approximately solving a sequence of well-chosen auxiliary problems, leading to faster convergence. One of the keys to achieve acceleration in theory and in practice is to solve these sub-problems with appropriate accuracy by using the right stopping criterion and the right warm-start strategy. We give practical guidelines to use Catalyst and present a comprehensive analysis of its global complexity. We show that Catalyst applies to a large class of algorithms, including gradient descent, block coordinate descent, incremental algorithms such as SAG, SAGA, SDCA, SVRG, MISO/Finito, and their proximal variants. For all of these methods, we establish faster rates using the Catalyst acceleration, for strongly convex and non-strongly convex objectives. We conclude with extensive experiments showing that acceleration is useful in practice, especially for ill-conditioned problems.
Submission history
From: Julien Mairal [view email][v1] Fri, 15 Dec 2017 13:05:18 GMT (840kb,D)
[v2] Tue, 19 Jun 2018 07:05:12 GMT (810kb,D)
Link back to: arXiv, form interface, contact.