We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Optimizing Optimizers: Regret-optimal gradient descent algorithms

Abstract: The need for fast and robust optimization algorithms are of critical importance in all areas of machine learning. This paper treats the task of designing optimization algorithms as an optimal control problem. Using regret as a metric for an algorithm's performance, we derive the necessary and sufficient dynamics that regret-optimal algorithms must satisfy as a discrete-time difference equation. We study the existence, uniqueness and consistency of regret-optimal algorithms and derive bounds on rates of convergence to solutions of convex optimization problems. Though closed-form optimal dynamics cannot be obtained in general, we present fast numerical methods for approximating them, generating optimization algorithms which directly optimize their long-term regret. Lastly, these are benchmarked against commonly used optimization algorithms to demonstrate their effectiveness.
Comments: 12 pages body, 42 pages total, 2 figures
Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC)
MSC classes: 49K10, 49J10, 49M05, 49K27, 49J50, 65K10
ACM classes: F.2.0
Cite as: arXiv:2101.00041 [cs.LG]
  (or arXiv:2101.00041v1 [cs.LG] for this version)

Submission history

From: Philippe Casgrain [view email]
[v1] Thu, 31 Dec 2020 19:13:53 GMT (1201kb,D)
[v2] Tue, 19 Jan 2021 22:50:56 GMT (812kb,D)

Link back to: arXiv, form interface, contact.