References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: PromptCap: Prompt-Guided Task-Aware Image Captioning
(Submitted on 15 Nov 2022 (this version), latest version 17 Aug 2023 (v4))
Abstract: Image captioning aims to describe an image with a natural language sentence, allowing powerful language models to understand images. The framework of combining image captioning with language models has been successful on various vision-language tasks. However, an image contains much more information than a single sentence, leading to underspecification of which visual entities should be described in the caption sentence. For example, when performing visual questioning answering (VQA), generic image captions often miss visual details that are essential for the language model to answer correctly. To address this challenge, we propose PromptCap, a captioning model that takes a natural-language prompt to control the contents of the generated caption. The prompt contains a question that the caption should help to answer, and also supports taking auxiliary text inputs such as scene text within the image itself. To finetune a general image caption model for prompt-guided captioning, we propose a pipeline to synthesize and filter training examples with GPT-3 and existing VQA datasets. For evaluation, we start with an existing pipeline in which a language model is prompted with image captions to carry out VQA. With the same language model, a higher QA accuracy shows that our generated captions are more relevant to the question prompts. PromptCap outperforms generic captions by a large margin on a variety of VQA tasks and achieves the state-of-the-art accuracy of 58.8 % on OK-VQA and 58.0 % on A-OKVQA. Zero-shot experiments on WebQA show that PromptCap generalizes well to unseen domains.
Submission history
From: Yushi Hu [view email][v1] Tue, 15 Nov 2022 19:07:53 GMT (15510kb,D)
[v2] Tue, 21 Mar 2023 12:10:18 GMT (5838kb,D)
[v3] Tue, 28 Mar 2023 11:14:23 GMT (5838kb,D)
[v4] Thu, 17 Aug 2023 21:43:24 GMT (6155kb,D)
Link back to: arXiv, form interface, contact.