References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Counterfactual VQA: A Cause-Effect Look at Language Bias
(Submitted on 8 Jun 2020 (v1), last revised 1 Apr 2021 (this version, v4))
Abstract: VQA models may tend to rely on language bias as a shortcut and thus fail to sufficiently learn the multi-modal knowledge from both vision and language. Recent debiasing methods proposed to exclude the language prior during inference. However, they fail to disentangle the "good" language context and "bad" language bias from the whole. In this paper, we investigate how to mitigate language bias in VQA. Motivated by causal effects, we proposed a novel counterfactual inference framework, which enables us to capture the language bias as the direct causal effect of questions on answers and reduce the language bias by subtracting the direct language effect from the total causal effect. Experiments demonstrate that our proposed counterfactual inference framework 1) is general to various VQA backbones and fusion strategies, 2) achieves competitive performance on the language-bias sensitive VQA-CP dataset while performs robustly on the balanced VQA v2 dataset without any augmented data. The code is available at this https URL
Submission history
From: Yulei Niu [view email][v1] Mon, 8 Jun 2020 01:49:27 GMT (879kb,D)
[v2] Mon, 15 Jun 2020 16:08:46 GMT (1504kb,D)
[v3] Mon, 28 Dec 2020 10:35:08 GMT (3223kb,D)
[v4] Thu, 1 Apr 2021 16:15:36 GMT (9062kb,D)
Link back to: arXiv, form interface, contact.