Current browse context:
cs.CV
Change to browse by:
References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Contrastive Learning for Weakly Supervised Phrase Grounding
(Submitted on 17 Jun 2020 (v1), last revised 5 Aug 2020 (this version, v3))
Abstract: Phrase grounding, the problem of associating image regions to caption words, is a crucial component of vision-language tasks. We show that phrase grounding can be learned by optimizing word-region attention to maximize a lower bound on mutual information between images and caption words. Given pairs of images and captions, we maximize compatibility of the attention-weighted regions and the words in the corresponding caption, compared to non-corresponding pairs of images and captions. A key idea is to construct effective negative captions for learning through language model guided word substitutions. Training with our negatives yields a $\sim10\%$ absolute gain in accuracy over randomly-sampled negatives from the training data. Our weakly supervised phrase grounding model trained on COCO-Captions shows a healthy gain of $5.7\%$ to achieve $76.7\%$ accuracy on Flickr30K Entities benchmark.
Submission history
From: Arash Vahdat [view email][v1] Wed, 17 Jun 2020 15:00:53 GMT (4162kb,D)
[v2] Sat, 25 Jul 2020 04:11:42 GMT (10948kb,D)
[v3] Wed, 5 Aug 2020 21:53:38 GMT (10948kb,D)
Link back to: arXiv, form interface, contact.