Current browse context:
cs.CV
Change to browse by:
References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Bridge-Tower: Building Bridges Between Encoders in Vision-Language Representation Learning
(Submitted on 17 Jun 2022 (this version), latest version 27 Mar 2024 (v6))
Abstract: Vision-Language (VL) models with the Two-Tower architecture have dominated visual-language representation learning in recent years. Current VL models either use lightweight uni-modal encoders and learn to extract, align and fuse both modalities simultaneously in a cross-modal encoder, or feed the last-layer uni-modal features directly into the top cross-modal encoder, ignoring the semantic information at the different levels in the deep uni-modal encoders. Both approaches possibly restrict vision-language representation learning and limit model performance. In this paper, we introduce multiple bridge layers that build a connection between the top layers of uni-modal encoders and each layer of the cross-modal encoder. This enables comprehensive bottom-up interactions between visual and textual representations at different semantic levels, resulting in more effective cross-modal alignment and fusion. Our proposed Bridge-Tower, pre-trained with only $4$M images, achieves state-of-the-art performance on various downstream vision-language tasks. On the VQAv2 test-std set, Bridge-Tower achieves an accuracy of $78.73\%$, outperforming the previous state-of-the-art METER model by $1.09\%$ with the same pre-training data and almost no additional parameters and computational cost. Notably, when further scaling the model, Bridge-Tower achieves an accuracy of $81.15\%$, surpassing models that are pre-trained on orders-of-magnitude larger datasets. Code is available at this https URL
Submission history
From: Xiao Xu [view email][v1] Fri, 17 Jun 2022 09:42:35 GMT (964kb,D)
[v2] Thu, 24 Nov 2022 12:20:24 GMT (1349kb,D)
[v3] Thu, 26 Jan 2023 14:16:34 GMT (1348kb,D)
[v4] Thu, 2 Feb 2023 16:22:23 GMT (1348kb,D)
[v5] Fri, 9 Jun 2023 12:36:33 GMT (1387kb,D)
[v6] Wed, 27 Mar 2024 03:53:23 GMT (1074kb,D)
Link back to: arXiv, form interface, contact.