Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Improved Learning Rates for Stochastic Optimization: Two Theoretical Viewpoints
(Submitted on 19 Jul 2021 (v1), last revised 25 Jul 2021 (this version, v2))
Abstract: Generalization performance of stochastic optimization stands a central place in learning theory. In this paper, we investigate the excess risk performance and towards improved learning rates for two popular approaches of stochastic optimization: empirical risk minimization (ERM) and stochastic gradient descent (SGD). Although there exists plentiful generalization analysis of ERM and SGD for supervised learning, current theoretical understandings of ERM and SGD either have stronger assumptions in convex learning, e.g., strong convexity, or show slow rates and less studied in nonconvex learning. Motivated by these problems, we aim to provide improved rates under milder assumptions in convex learning and derive faster rates in nonconvex learning. It is notable that our analysis span two popular theoretical viewpoints: \emph{stability} and \emph{uniform convergence}. Specifically, in stability regime, we present high probability learning rates of order $\mathcal{O} (1/n)$ w.r.t. the sample size $n$ for ERM and SGD with milder assumptions in convex learning and similar high probability rates of order $\mathcal{O} (1/n)$ in nonconvex learning, rather than in expectation. Furthermore, this type of learning rate is improved to faster order $\mathcal{O} (1/n^2)$ in uniform convergence regime. To our best knowledge, for ERM and SGD, the learning rates presented in this paper are all state-of-the-art.
Submission history
From: Shaojie Li [view email][v1] Mon, 19 Jul 2021 08:46:14 GMT (571kb)
[v2] Sun, 25 Jul 2021 14:45:28 GMT (71kb)
Link back to: arXiv, form interface, contact.