We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computer Vision and Pattern Recognition

Title: Gender and Racial Bias in Visual Question Answering Datasets

Abstract: Vision-and-language tasks have increasingly drawn more attention as a means to evaluate human-like reasoning in machine learning models. A popular task in the field is visual question answering (VQA), which aims to answer questions about images. However, VQA models have been shown to exploit language bias by learning the statistical correlations between questions and answers without looking into the image content: e.g., questions about the color of a banana are answered with yellow, even if the banana in the image is green. If societal bias (e.g., sexism, racism, ableism, etc.) is present in the training data, this problem may be causing VQA models to learn harmful stereotypes. For this reason, we investigate gender and racial bias in five VQA datasets. In our analysis, we find that the distribution of answers is highly different between questions about women and men, as well as the existence of detrimental gender-stereotypical samples. Likewise, we identify that specific race-related attributes are underrepresented, whereas potentially discriminatory samples appear in the analyzed datasets. Our findings suggest that there are dangers associated to using VQA datasets without considering and dealing with the potentially harmful stereotypes. We conclude the paper by proposing solutions to alleviate the problem before, during, and after the dataset collection process.
Comments: ACM Conference on Fairness, Accountability, and Transparency (FAccT 2022)
Subjects: Computer Vision and Pattern Recognition (cs.CV); Computers and Society (cs.CY)
DOI: 10.1145/3531146.3533184
Cite as: arXiv:2205.08148 [cs.CV]
  (or arXiv:2205.08148v3 [cs.CV] for this version)

Submission history

From: Yusuke Hirota [view email]
[v1] Tue, 17 May 2022 07:33:24 GMT (3997kb,D)
[v2] Wed, 18 May 2022 08:47:20 GMT (3997kb,D)
[v3] Fri, 3 Jun 2022 06:36:16 GMT (3974kb,D)

Link back to: arXiv, form interface, contact.