Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Stochastic Training is Not Necessary for Generalization
(Submitted on 29 Sep 2021 (v1), last revised 19 Apr 2022 (this version, v2))
Abstract: It is widely believed that the implicit regularization of SGD is fundamental to the impressive generalization behavior we observe in neural networks. In this work, we demonstrate that non-stochastic full-batch training can achieve comparably strong performance to SGD on CIFAR-10 using modern architectures. To this end, we show that the implicit regularization of SGD can be completely replaced with explicit regularization even when comparing against a strong and well-researched baseline. Our observations indicate that the perceived difficulty of full-batch training may be the result of its optimization properties and the disproportionate time and effort spent by the ML community tuning optimizers and hyperparameters for small-batch training.
Submission history
From: Jonas Geiping [view email][v1] Wed, 29 Sep 2021 00:50:00 GMT (553kb,D)
[v2] Tue, 19 Apr 2022 22:01:13 GMT (1007kb,D)
Link back to: arXiv, form interface, contact.