References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Look at What I'm Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos
(Submitted on 20 Oct 2021 (v1), last revised 2 Dec 2021 (this version, v2))
Abstract: We introduce the task of spatially localizing narrated interactions in videos. Key to our approach is the ability to learn to spatially localize interactions with self-supervision on a large corpus of videos with accompanying transcribed narrations. To achieve this goal, we propose a multilayer cross-modal attention network that enables effective optimization of a contrastive loss during training. We introduce a divided strategy that alternates between computing inter- and intra-modal attention across the visual and natural language modalities, which allows effective training via directly contrasting the two modalities' representations. We demonstrate the effectiveness of our approach by self-training on the HowTo100M instructional video dataset and evaluating on a newly collected dataset of localized described interactions in the YouCook2 dataset. We show that our approach outperforms alternative baselines, including shallow co-attention and full cross-modal attention. We also apply our approach to grounding phrases in images with weak supervision on Flickr30K and show that stacking multiple attention layers is effective and, when combined with a word-to-region loss, achieves state of the art on recall-at-one and pointing hand accuracies.
Submission history
From: Reuben Tan [view email][v1] Wed, 20 Oct 2021 14:45:13 GMT (8228kb,D)
[v2] Thu, 2 Dec 2021 16:55:56 GMT (2206kb,D)
Link back to: arXiv, form interface, contact.