Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Assessing Generalization of SGD via Disagreement
(Submitted on 25 Jun 2021 (v1), last revised 15 May 2022 (this version, v2))
Abstract: We empirically show that the test error of deep networks can be estimated by simply training the same architecture on the same training set but with a different run of Stochastic Gradient Descent (SGD), and measuring the disagreement rate between the two networks on unlabeled test data. This builds on -- and is a stronger version of -- the observation in Nakkiran & Bansal '20, which requires the second run to be on an altogether fresh training set. We further theoretically show that this peculiar phenomenon arises from the \emph{well-calibrated} nature of \emph{ensembles} of SGD-trained models. This finding not only provides a simple empirical measure to directly predict the test error using unlabeled test data, but also establishes a new conceptual connection between generalization and calibration.
Submission history
From: Vaishnavh Nagarajan [view email][v1] Fri, 25 Jun 2021 17:53:09 GMT (1550kb,D)
[v2] Sun, 15 May 2022 20:53:46 GMT (1172kb,D)
Link back to: arXiv, form interface, contact.