We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Understanding Gradient Descent on Edge of Stability in Deep Learning

Abstract: Deep learning experiments by Cohen et al. [2021] using deterministic Gradient Descent (GD) revealed an Edge of Stability (EoS) phase when learning rate (LR) and sharpness (i.e., the largest eigenvalue of Hessian) no longer behave as in traditional optimization. Sharpness stabilizes around $2/$LR and loss goes up and down across iterations, yet still with an overall downward trend. The current paper mathematically analyzes a new mechanism of implicit regularization in the EoS phase, whereby GD updates due to non-smooth loss landscape turn out to evolve along some deterministic flow on the manifold of minimum loss. This is in contrast to many previous results about implicit bias either relying on infinitesimal updates or noise in gradient. Formally, for any smooth function $L$ with certain regularity condition, this effect is demonstrated for (1) Normalized GD, i.e., GD with a varying LR $\eta_t =\frac{\eta}{|| \nabla L(x(t)) ||}$ and loss $L$; (2) GD with constant LR and loss $\sqrt{L- \min_x L(x)}$. Both provably enter the Edge of Stability, with the associated flow on the manifold minimizing $\lambda_{1}(\nabla^2 L)$. The above theoretical results have been corroborated by an experimental study.
Comments: This paper has been accepted for conference proceedings in the 39th International Conference on Machine Learning (ICML), 2022
Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC)
Cite as: arXiv:2205.09745 [cs.LG]
  (or arXiv:2205.09745v2 [cs.LG] for this version)

Submission history

From: Abhishek Panigrahi [view email]
[v1] Thu, 19 May 2022 17:57:01 GMT (621kb,D)
[v2] Mon, 27 Jun 2022 18:09:40 GMT (2040kb,D)

Link back to: arXiv, form interface, contact.