Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima
(Submitted on 10 Feb 2020 (v1), last revised 15 Jan 2021 (this version, v14))
Abstract: Stochastic Gradient Descent (SGD) and its variants are mainstream methods for training deep networks in practice. SGD is known to find a flat minimum that often generalizes well. However, it is mathematically unclear how deep learning can select a flat minimum among so many minima. To answer the question quantitatively, we develop a density diffusion theory (DDT) to reveal how minima selection quantitatively depends on the minima sharpness and the hyperparameters. To the best of our knowledge, we are the first to theoretically and empirically prove that, benefited from the Hessian-dependent covariance of stochastic gradient noise, SGD favors flat minima exponentially more than sharp minima, while Gradient Descent (GD) with injected white noise favors flat minima only polynomially more than sharp minima. We also reveal that either a small learning rate or large-batch training requires exponentially many iterations to escape from minima in terms of the ratio of the batch size and learning rate. Thus, large-batch training cannot search flat minima efficiently in a realistic computational time.
Submission history
From: Zeke Xie [view email][v1] Mon, 10 Feb 2020 02:04:49 GMT (2689kb)
[v2] Tue, 18 Feb 2020 06:24:38 GMT (2690kb)
[v3] Thu, 20 Feb 2020 04:40:24 GMT (2690kb)
[v4] Wed, 26 Feb 2020 08:27:12 GMT (2690kb)
[v5] Thu, 5 Mar 2020 12:04:23 GMT (2690kb)
[v6] Tue, 14 Apr 2020 10:51:51 GMT (5760kb)
[v7] Mon, 4 May 2020 08:11:19 GMT (5756kb)
[v8] Thu, 21 May 2020 00:54:13 GMT (3011kb,D)
[v9] Mon, 22 Jun 2020 03:52:54 GMT (2427kb,D)
[v10] Mon, 29 Jun 2020 05:27:27 GMT (2427kb,D)
[v11] Sat, 4 Jul 2020 04:54:20 GMT (2427kb,D)
[v12] Sat, 26 Sep 2020 11:36:52 GMT (2910kb,D)
[v13] Tue, 24 Nov 2020 05:12:13 GMT (6395kb,D)
[v14] Fri, 15 Jan 2021 14:57:46 GMT (6409kb,D)
Link back to: arXiv, form interface, contact.