Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Online to Offline Conversions, Universality and Adaptive Minibatch Sizes
(Submitted on 30 May 2017 (v1), last revised 31 May 2017 (this version, v2))
Abstract: We present an approach towards convex optimization that relies on a novel scheme which converts online adaptive algorithms into offline methods. In the offline optimization setting, our derived methods are shown to obtain favourable adaptive guarantees which depend on the harmonic sum of the queried gradients. We further show that our methods implicitly adapt to the objective's structure: in the smooth case fast convergence rates are ensured without any prior knowledge of the smoothness parameter, while still maintaining guarantees in the non-smooth setting. Our approach has a natural extension to the stochastic setting, resulting in a lazy version of SGD (stochastic GD), where minibathces are chosen \emph{adaptively} depending on the magnitude of the gradients. Thus providing a principled approach towards choosing minibatch sizes.
Submission history
From: Kfir Levy Yehuda [view email][v1] Tue, 30 May 2017 08:08:36 GMT (1713kb,D)
[v2] Wed, 31 May 2017 07:22:55 GMT (1713kb,D)
Link back to: arXiv, form interface, contact.