Current browse context:
cond-mat.dis-nn
Change to browse by:
References & Citations
Condensed Matter > Disordered Systems and Neural Networks
Title: Analytical Study of Momentum-Based Acceleration Methods in Paradigmatic High-Dimensional Non-Convex Problems
(Submitted on 23 Feb 2021 (v1), last revised 27 Oct 2021 (this version, v4))
Abstract: The optimization step in many machine learning problems rarely relies on vanilla gradient descent but it is common practice to use momentum-based accelerated methods. Despite these algorithms being widely applied to arbitrary loss functions, their behaviour in generically non-convex, high dimensional landscapes is poorly understood. In this work, we use dynamical mean field theory techniques to describe analytically the average dynamics of these methods in a prototypical non-convex model: the (spiked) matrix-tensor model. We derive a closed set of equations that describe the behaviour of heavy-ball momentum and Nesterov acceleration in the infinite dimensional limit. By numerical integration of these equations, we observe that these methods speed up the dynamics but do not improve the algorithmic threshold with respect to gradient descent in the spiked model.
Submission history
From: Stefano Sarao Mannelli [view email][v1] Tue, 23 Feb 2021 15:30:57 GMT (10068kb,D)
[v2] Wed, 24 Feb 2021 09:40:16 GMT (10068kb,D)
[v3] Thu, 11 Mar 2021 14:03:30 GMT (10068kb,D)
[v4] Wed, 27 Oct 2021 22:07:37 GMT (9365kb,D)
Link back to: arXiv, form interface, contact.