References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Video Understanding as Machine Translation
(Submitted on 12 Jun 2020 (v1), last revised 17 Sep 2020 (this version, v2))
Abstract: With the advent of large-scale multimodal video datasets, especially sequences with audio or transcribed speech, there has been a growing interest in self-supervised learning of video representations. Most prior work formulates the objective as a contrastive metric learning problem between the modalities. To enable effective learning, however, these strategies require a careful selection of positive and negative samples often combined with hand-designed curriculum policies. In this work we remove the need for negative sampling by taking a generative modeling approach that poses the objective as a translation problem between modalities. Such a formulation allows us to tackle a wide variety of downstream video understanding tasks by means of a single unified framework, without the need for large batches of negative samples common in contrastive metric learning. We experiment with the large-scale HowTo100M dataset for training, and report performance gains over the state-of-the-art on several downstream tasks including video classification (EPIC-Kitchens), question answering (TVQA), captioning (TVC, YouCook2, and MSR-VTT), and text-based clip retrieval (YouCook2 and MSR-VTT).
Submission history
From: Bruno Korbar [view email][v1] Fri, 12 Jun 2020 14:07:04 GMT (6641kb,D)
[v2] Thu, 17 Sep 2020 19:41:31 GMT (0kb,I)
Link back to: arXiv, form interface, contact.