Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: The Value of Out-of-Distribution Data
(Submitted on 23 Aug 2022 (v1), revised 6 Oct 2022 (this version, v2), latest version 13 Jul 2023 (v5))
Abstract: More data is expected to help us generalize to a task. But real datasets can contain out-of-distribution (OOD) data; this can come in the form of heterogeneity such as intra-class variability but also in the form of temporal shifts or concept drifts. We demonstrate a counter-intuitive phenomenon for such problems: generalization error of the task can be a non-monotonic function of the number of OOD samples; a small number of OOD samples can improve generalization but if the number of OOD samples is beyond a threshold, then the generalization error can deteriorate. We also show that if we know which samples are OOD, then using a weighted objective between the target and OOD samples ensures that the generalization error decreases monotonically. We demonstrate and analyze this phenomenon using linear classifiers on synthetic datasets and medium-sized neural networks on vision benchmarks such as MNIST, CIFAR-10, CINIC-10, PACS, and DomainNet, and observe the effect data augmentation, hyperparameter optimization, and pre-training have on this behavior.
Submission history
From: Ashwin De Silva [view email][v1] Tue, 23 Aug 2022 13:41:01 GMT (2907kb)
[v2] Thu, 6 Oct 2022 10:12:00 GMT (4119kb,D)
[v3] Thu, 2 Feb 2023 03:31:21 GMT (4163kb,D)
[v4] Mon, 10 Jul 2023 09:15:22 GMT (685kb,D)
[v5] Thu, 13 Jul 2023 10:02:22 GMT (685kb,D)
Link back to: arXiv, form interface, contact.