Current browse context:
cs.CV
Change to browse by:
References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: ECLIPSE: Efficient Long-range Video Retrieval using Sight and Sound
(Submitted on 6 Apr 2022 (v1), last revised 2 Aug 2022 (this version, v3))
Abstract: We introduce an audiovisual method for long-range text-to-video retrieval. Unlike previous approaches designed for short video retrieval (e.g., 5-15 seconds in duration), our approach aims to retrieve minute-long videos that capture complex human actions. One challenge of standard video-only approaches is the large computational cost associated with processing hundreds of densely extracted frames from such long videos. To address this issue, we propose to replace parts of the video with compact audio cues that succinctly summarize dynamic audio events and are cheap to process. Our method, named ECLIPSE (Efficient CLIP with Sound Encoding), adapts the popular CLIP model to an audiovisual video setting, by adding a unified audiovisual transformer block that captures complementary cues from the video and audio streams. In addition to being 2.92x faster and 2.34x memory-efficient than long-range video-only approaches, our method also achieves better text-to-video retrieval accuracy on several diverse long-range video datasets such as ActivityNet, QVHighlights, YouCook2, DiDeMo and Charades.
Submission history
From: Yan-Bo Lin [view email][v1] Wed, 6 Apr 2022 14:43:42 GMT (4459kb,D)
[v2] Thu, 21 Jul 2022 07:52:00 GMT (7207kb,D)
[v3] Tue, 2 Aug 2022 05:57:20 GMT (7207kb,D)
Link back to: arXiv, form interface, contact.