Current browse context:
cs.CV
Change to browse by:
References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: VoP: Text-Video Co-operative Prompt Tuning for Cross-Modal Retrieval
(Submitted on 23 Nov 2022 (v1), last revised 22 Mar 2023 (this version, v3))
Abstract: Many recent studies leverage the pre-trained CLIP for text-video cross-modal retrieval by tuning the backbone with additional heavy modules, which not only brings huge computational burdens with much more parameters, but also leads to the knowledge forgetting from upstream models. In this work, we propose the VoP: Text-Video Co-operative Prompt Tuning for efficient tuning on the text-video retrieval task. The proposed VoP is an end-to-end framework with both video & text prompts introducing, which can be regarded as a powerful baseline with only 0.1% trainable parameters. Further, based on the spatio-temporal characteristics of videos, we develop three novel video prompt mechanisms to improve the performance with different scales of trainable parameters. The basic idea of the VoP enhancement is to model the frame position, frame context, and layer function with specific trainable prompts, respectively. Extensive experiments show that compared to full fine-tuning, the enhanced VoP achieves a 1.4% average R@1 gain across five text-video retrieval benchmarks with 6x less parameter overhead. The code will be available at this https URL
Submission history
From: Siteng Huang [view email][v1] Wed, 23 Nov 2022 08:20:29 GMT (763kb,D)
[v2] Wed, 8 Mar 2023 06:31:05 GMT (757kb,D)
[v3] Wed, 22 Mar 2023 02:36:52 GMT (758kb,D)
Link back to: arXiv, form interface, contact.