Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification
(Submitted on 10 Jun 2020 (v1), last revised 9 Nov 2021 (this version, v2))
Abstract: We analyze in a closed form the learning dynamics of stochastic gradient descent (SGD) for a single-layer neural network classifying a high-dimensional Gaussian mixture where each cluster is assigned one of two labels. This problem provides a prototype of a non-convex loss landscape with interpolating regimes and a large generalization gap. We define a particular stochastic process for which SGD can be extended to a continuous-time limit that we call stochastic gradient flow. In the full-batch limit, we recover the standard gradient flow. We apply dynamical mean-field theory from statistical physics to track the dynamics of the algorithm in the high-dimensional limit via a self-consistent stochastic process. We explore the performance of the algorithm as a function of the control parameters shedding light on how it navigates the loss landscape.
Submission history
From: Francesca Mignacco [view email][v1] Wed, 10 Jun 2020 22:49:41 GMT (884kb,D)
[v2] Tue, 9 Nov 2021 13:33:26 GMT (976kb,D)
Link back to: arXiv, form interface, contact.