Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: The Value of Out-of-Distribution Data
(Submitted on 23 Aug 2022 (v1), last revised 13 Jul 2023 (this version, v5))
Abstract: We expect the generalization error to improve with more samples from a similar task, and to deteriorate with more samples from an out-of-distribution (OOD) task. In this work, we show a counter-intuitive phenomenon: the generalization error of a task can be a non-monotonic function of the number of OOD samples. As the number of OOD samples increases, the generalization error on the target task improves before deteriorating beyond a threshold. In other words, there is value in training on small amounts of OOD data. We use Fisher's Linear Discriminant on synthetic datasets and deep networks on computer vision benchmarks such as MNIST, CIFAR-10, CINIC-10, PACS and DomainNet to demonstrate and analyze this phenomenon. In the idealistic setting where we know which samples are OOD, we show that these non-monotonic trends can be exploited using an appropriately weighted objective of the target and OOD empirical risk. While its practical utility is limited, this does suggest that if we can detect OOD samples, then there may be ways to benefit from them. When we do not know which samples are OOD, we show how a number of go-to strategies such as data-augmentation, hyper-parameter optimization, and pre-training are not enough to ensure that the target generalization error does not deteriorate with the number of OOD samples in the dataset.
Submission history
From: Ashwin De Silva [view email][v1] Tue, 23 Aug 2022 13:41:01 GMT (2907kb)
[v2] Thu, 6 Oct 2022 10:12:00 GMT (4119kb,D)
[v3] Thu, 2 Feb 2023 03:31:21 GMT (4163kb,D)
[v4] Mon, 10 Jul 2023 09:15:22 GMT (685kb,D)
[v5] Thu, 13 Jul 2023 10:02:22 GMT (685kb,D)
Link back to: arXiv, form interface, contact.