Current browse context:
cs.CL
Change to browse by:
References & Citations
Computer Science > Computation and Language
Title: Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalization
(Submitted on 24 May 2022 (v1), last revised 1 Apr 2023 (this version, v2))
Abstract: Vision-and-language (V&L) models pretrained on large-scale multimodal data have demonstrated strong performance on various tasks such as image captioning and visual question answering (VQA). The quality of such models is commonly assessed by measuring their performance on unseen data that typically comes from the same distribution as the training data. However, when evaluated under out-of-distribution (out-of-dataset) settings for VQA, we observe that these models exhibit poor generalization. We comprehensively evaluate two pretrained V&L models under different settings (i.e. classification and open-ended text generation) by conducting cross-dataset evaluations. We find that these models tend to learn to solve the benchmark, rather than learning the high-level skills required by the VQA task. We also find that in most cases generative models are less susceptible to shifts in data distribution compared to discriminative ones, and that multimodal pretraining is generally helpful for OOD generalization. Finally, we revisit assumptions underlying the use of automatic VQA evaluation metrics, and empirically show that their stringent nature repeatedly penalizes models for correct responses.
Submission history
From: Emanuele Bugliarello [view email][v1] Tue, 24 May 2022 16:44:45 GMT (2891kb,D)
[v2] Sat, 1 Apr 2023 07:07:44 GMT (4904kb,D)
Link back to: arXiv, form interface, contact.