We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

math.OC

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Mathematics > Optimization and Control

Title: First order online optimisation using forward gradients in over-parameterised systems

Abstract: The success of deep learning over the past decade mainly relies on gradient-based optimisation and backpropagation. This paper focuses on analysing the performance of first-order gradient-based optimisation algorithms, gradient descent and proximal gradient, with time-varying non-convex cost function under (proximal) Polyak-{\L}ojasiewicz condition. Specifically, we focus on using the forward mode of automatic differentiation to compute gradients in the fast-changing problems where calculating gradients using the backpropagation algorithm is either impossible or inefficient. Upper bounds for tracking and asymptotic errors are derived for various cases, showing the linear convergence to a solution or a neighbourhood of an optimal solution, where the convergence rate decreases with the increase in the dimension of the problem. We show that for a solver with constraints on computing resources, the number of forward gradient iterations at each step can be a design parameter that trades off between the tracking performance and computing constraints.
Comments: 18 pages, 1 figure
Subjects: Optimization and Control (math.OC)
Cite as: arXiv:2211.15825 [math.OC]
  (or arXiv:2211.15825v3 [math.OC] for this version)

Submission history

From: Behnam Mafakheri [view email]
[v1] Mon, 28 Nov 2022 23:22:40 GMT (332kb,D)
[v2] Thu, 1 Dec 2022 05:13:57 GMT (333kb,D)
[v3] Tue, 6 Dec 2022 22:40:01 GMT (333kb,D)

Link back to: arXiv, form interface, contact.