References & Citations
Computer Science > Computation and Language
Title: Measuring the Robustness of NLP Models to Domain Shifts
(Submitted on 31 May 2023 (v1), last revised 28 Jan 2024 (this version, v4))
Abstract: Existing research on Domain Robustness (DR) suffers from disparate setups, lack of task variety, and scarce research on recent capabilities such as few-shot learning. Furthermore, we claim that the common practice of measuring DR might further obscure the picture. Current research focuses on challenge sets and relies solely on the Source Drop (SD): Using the source in-domain performance as a reference point for degradation. However, the Target Drop (TD), which measures degradation from the target in-domain performance, should be used as a complementary point of view. In this study, we developed a benchmark comprised of seven NLP tasks, including classification, QA, and generation. Our benchmark focuses on natural topical domain shifts and enables measuring both the SD and the TD. Our comprehensive study, involving over 14,000 domain shifts across 18 fine-tuned and few-shot models, shows that both model types suffer from drops upon domain shifts. While fine-tuned models excel in-domain, few-shot LLMs often surpass them cross-domain, showing better robustness. In addition, we found that a large SD can be explained by shifting to a harder domain rather than by a genuine DR challenge. Thus, the TD is a more reliable metric for assessing DR.
Submission history
From: Nitay Calderon [view email][v1] Wed, 31 May 2023 20:25:08 GMT (175kb,D)
[v2] Sat, 1 Jul 2023 18:05:19 GMT (1017kb,D)
[v3] Fri, 19 Jan 2024 13:05:04 GMT (1164kb,D)
[v4] Sun, 28 Jan 2024 13:06:38 GMT (1230kb,D)
Link back to: arXiv, form interface, contact.