Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Neural Network Weights Do Not Converge to Stationary Points: An Invariant Measure Perspective
(Submitted on 12 Oct 2021 (v1), last revised 17 Jun 2022 (this version, v2))
Abstract: This work examines the deep disconnect between existing theoretical analyses of gradient-based algorithms and the practice of training deep neural networks. Specifically, we provide numerical evidence that in large-scale neural network training (e.g., ImageNet + ResNet101, and WT103 + TransformerXL models), the neural network's weights do not converge to stationary points where the gradient of the loss is zero. Remarkably, however, we observe that even though the weights do not converge to stationary points, the progress in minimizing the loss function halts and training loss stabilizes. Inspired by this observation, we propose a new perspective based on ergodic theory of dynamical systems to explain it. Rather than studying the evolution of weights, we study the evolution of the distribution of weights. We prove convergence of the distribution of weights to an approximate invariant measure, thereby explaining how the training loss can stabilize without weights necessarily converging to stationary points. We further discuss how this perspective can better align optimization theory with empirical observations in machine learning practice.
Submission history
From: Jingzhao Zhang [view email][v1] Tue, 12 Oct 2021 18:12:23 GMT (1422kb)
[v2] Fri, 17 Jun 2022 10:01:32 GMT (1584kb,D)
Link back to: arXiv, form interface, contact.