Current browse context:
cs.CL
Change to browse by:
References & Citations
Computer Science > Computation and Language
Title: Weakly-supervised VisualBERT: Pre-training without Parallel Images and Captions
(Submitted on 24 Oct 2020 (this version), latest version 11 Apr 2021 (v2))
Abstract: Pre-trained contextual vision-and-language (V&L) models have brought impressive performance improvement on various benchmarks. However, the paired text-image data required for pre-training are hard to collect and scale up. We investigate if a strong V&L representation model can be learned without text-image pairs. We propose Weakly-supervised VisualBERT with the key idea of conducting "mask-and-predict" pre-training on language-only and image-only corpora. Additionally, we introduce the object tags detected by an object recognition model as anchor points to bridge two modalities. Evaluation on four V&L benchmarks shows that Weakly-supervised VisualBERT achieves similar performance with a model pre-trained with paired data. Besides, pre-training on more image-only data further improves a model that already has access to aligned data, suggesting the possibility of utilizing billions of raw images available to enhance V&L models.
Submission history
From: Liunian Harold Li [view email][v1] Sat, 24 Oct 2020 08:17:54 GMT (7920kb,D)
[v2] Sun, 11 Apr 2021 23:54:25 GMT (4754kb,D)
Link back to: arXiv, form interface, contact.