Current browse context:
math.OC
Change to browse by:
References & Citations
Mathematics > Optimization and Control
Title: FLAG: Fast Linearly-Coupled Adaptive Gradient Method
(Submitted on 26 May 2016 (this version), latest version 11 Nov 2017 (v2))
Abstract: The celebrated Nesterov's accelerated gradient method offers great speed-ups compared to the classical gradient descend method as it attains the optimal first-order oracle complexity for smooth convex optimization. On the other hand, the popular AdaGrad algorithm competes with mirror descent under the best regularizer by adaptively scaling the gradient. Recently, it has been shown that the accelerated gradient descent can be viewed as a linear combination of gradient descent and mirror descent steps. Here, we draw upon these ideas and present a fast linearly-coupled adaptive gradient method (FLAG) as an accelerated version of AdaGrad, and show that our algorithm can indeed offer the best of both worlds. Like Nesterov's accelerated algorithm and its proximal variant, FISTA, our method has a convergence rate of $1/T^2$ after $T$ iterations. Like AdaGrad our method adaptively chooses a regularizer, in a way that performs almost as well as the best choice of regularizer in hindsight.
Submission history
From: Xiang Cheng [view email][v1] Thu, 26 May 2016 00:24:37 GMT (19kb)
[v2] Sat, 11 Nov 2017 20:20:17 GMT (2054kb,D)
Link back to: arXiv, form interface, contact.