We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

math.OC

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Mathematics > Optimization and Control

Title: On Constraints in First-Order Optimization: A View from Non-Smooth Dynamical Systems

Abstract: We introduce a class of first-order methods for smooth constrained optimization that are based on an analogy to non-smooth dynamical systems. Two distinctive features of our approach are that (i) projections or optimizations over the entire feasible set are avoided, in stark contrast to projected gradient methods or the Frank-Wolfe method, and (ii) iterates are allowed to become infeasible, which differs from active set or feasible direction methods, where the descent motion stops as soon as a new constraint is encountered. The resulting algorithmic procedure is simple to implement even when constraints are nonlinear, and is suitable for large-scale constrained optimization problems in which the feasible set fails to have a simple structure. The key underlying idea is that constraints are expressed in terms of velocities instead of positions, which has the algorithmic consequence that optimizations over feasible sets at each iteration are replaced with optimizations over local, sparse convex approximations. In particular, this means that at each iteration only constraints that are violated are taken into account. The result is a simplified suite of algorithms and an expanded range of possible applications in machine learning.
Comments: 47 pages, 11 figures
Subjects: Optimization and Control (math.OC); Machine Learning (cs.LG); Systems and Control (eess.SY)
Cite as: arXiv:2107.08225 [math.OC]
  (or arXiv:2107.08225v3 [math.OC] for this version)

Submission history

From: Michael Muehlebach [view email]
[v1] Sat, 17 Jul 2021 11:45:13 GMT (228kb,D)
[v2] Thu, 17 Mar 2022 13:35:08 GMT (245kb,D)
[v3] Sat, 5 Nov 2022 10:33:59 GMT (244kb,D)

Link back to: arXiv, form interface, contact.