References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: VL-BEiT: Generative Vision-Language Pretraining
(Submitted on 2 Jun 2022 (v1), last revised 3 Sep 2022 (this version, v2))
Abstract: We introduce a vision-language foundation model called VL-BEiT, which is a bidirectional multimodal Transformer learned by generative pretraining. Our minimalist solution conducts masked prediction on both monomodal and multimodal data with a shared Transformer. Specifically, we perform masked vision-language modeling on image-text pairs, masked language modeling on texts, and masked image modeling on images. VL-BEiT is learned from scratch with one unified pretraining task, one shared backbone, and one-stage training. Our method is conceptually simple and empirically effective. Experimental results show that VL-BEiT obtains strong results on various vision-language benchmarks, such as visual question answering, visual reasoning, and image-text retrieval. Moreover, our method learns transferable visual features, achieving competitive performance on image classification, and semantic segmentation.
Submission history
From: Li Dong [view email][v1] Thu, 2 Jun 2022 16:14:19 GMT (682kb,D)
[v2] Sat, 3 Sep 2022 14:18:55 GMT (682kb,D)
Link back to: arXiv, form interface, contact.