We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

math.OC

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Mathematics > Optimization and Control

Title: On the Oracle Complexity of Higher-Order Smooth Non-Convex Finite-Sum Optimization

Abstract: We prove lower bounds for higher-order methods in smooth non-convex finite-sum optimization. Our contribution is threefold: We first show that a deterministic algorithm cannot profit from the finite-sum structure of the objective, and that simulating a pth-order regularized method on the whole function by constructing exact gradient information is optimal up to constant factors. We further show lower bounds for randomized algorithms and compare them with the best known upper bounds. To address some gaps between the bounds, we propose a new second-order smoothness assumption that can be seen as an analogue of the first-order mean-squared smoothness assumption. We prove that it is sufficient to ensure state-of-the-art convergence guarantees, while allowing for a sharper lower bound.
Comments: Added missing upper bound assumption on n in Theorems 4.7 and 4.10
Subjects: Optimization and Control (math.OC); Data Structures and Algorithms (cs.DS); Machine Learning (cs.LG); Machine Learning (stat.ML)
Cite as: arXiv:2103.05138 [math.OC]
  (or arXiv:2103.05138v2 [math.OC] for this version)

Submission history

From: Nicolas Emmenegger [view email]
[v1] Mon, 8 Mar 2021 23:33:58 GMT (41kb,D)
[v2] Fri, 2 Jul 2021 13:23:36 GMT (44kb,D)

Link back to: arXiv, form interface, contact.