References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Caption supervision enables robust learners
(Submitted on 13 Oct 2022 (this version), latest version 8 Dec 2022 (v2))
Abstract: Vision language models like CLIP are robust to natural distribution shifts, in part because CLIP learns on unstructured data using a technique called caption supervision; the model inteprets image-linked texts as ground-truth labels. In a carefully controlled comparison study, we show that CNNs trained on a standard cross-entropy loss can also benefit from caption supervision, in some cases even more than VL models, on the same data. To facilitate future experiments with high-accuracy caption-supervised models, we introduce CaptionNet (this https URL), which includes a class-balanced, fully supervised dataset with over 50,000 new human-labeled ImageNet-compliant samples which includes web-scraped captions. In a series of experiments on CaptionNet, we show how the choice of loss function, data filtration and supervision strategy enable robust computer vision. We also provide the codebase necessary to reproduce our experiments at this https URL
Submission history
From: Benjamin Feuer [view email][v1] Thu, 13 Oct 2022 22:29:10 GMT (15283kb,D)
[v2] Thu, 8 Dec 2022 14:28:09 GMT (15391kb,D)
Link back to: arXiv, form interface, contact.