Title: Learning Visual Classifiers using Human-centric Annotations

Abstract: Many datasets contain human-centric annotations that are the result of humans applying their own subjective judgements on what to describe and what to ignore. Examples include image tags and keywords found on photo sharing sites, or in datasets containing image captions. In this paper, we explore the use of human-centric annotations for learning image classifiers. Due to human reporting bias, these annotations miss a significant amount of the information present in an image. We propose an algorithm to decouple the human reporting bias from the correct visually grounded labels. Our algorithm provides results that are highly interpretable for reporting "what's in the image" versus "what's worth saying." We show improvements over traditional learning algorithms for both image classification and image captioning, and evaluate the algorithm's efficacy along a variety of metrics and datasets, including MS COCO and Yahoo Flickr 100M.