Current browse context:
math.OC
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Momentum via Primal Averaging: Theoretical Insights and Learning Rate Schedules for Non-Convex Optimization
(Submitted on 1 Oct 2020 (v1), last revised 1 Jun 2021 (this version, v4))
Abstract: Momentum methods are now used pervasively within the machine learning community for training non-convex models such as deep neural networks. Empirically, they out perform traditional stochastic gradient descent (SGD) approaches. In this work we develop a Lyapunov analysis of SGD with momentum (SGD+M), by utilizing a equivalent rewriting of the method known as the stochastic primal averaging (SPA) form. This analysis is much tighter than previous theory in the non-convex case, and due to this we are able to give precise insights into when SGD+M may out-perform SGD, and what hyper-parameter schedules will work and why.
Submission history
From: Aaron Defazio [view email][v1] Thu, 1 Oct 2020 13:46:32 GMT (225kb,D)
[v2] Tue, 3 Nov 2020 22:05:04 GMT (225kb,D)
[v3] Tue, 23 Feb 2021 19:49:57 GMT (218kb,D)
[v4] Tue, 1 Jun 2021 17:53:38 GMT (287kb,D)
Link back to: arXiv, form interface, contact.