Current browse context:
cs.CV
Change to browse by:
References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Show, Recall, and Tell: Image Captioning with Recall Mechanism
(Submitted on 15 Jan 2020 (v1), last revised 12 Mar 2021 (this version, v3))
Abstract: Generating natural and accurate descriptions in image cap-tioning has always been a challenge. In this paper, we pro-pose a novel recall mechanism to imitate the way human con-duct captioning. There are three parts in our recall mecha-nism : recall unit, semantic guide (SG) and recalled-wordslot (RWS). Recall unit is a text-retrieval module designedto retrieve recalled words for images. SG and RWS are de-signed for the best use of recalled words. SG branch cangenerate a recalled context, which can guide the process ofgenerating caption. RWS branch is responsible for copyingrecalled words to the caption. Inspired by pointing mecha-nism in text summarization, we adopt a soft switch to balancethe generated-word probabilities between SG and RWS. Inthe CIDEr optimization step, we also introduce an individualrecalled-word reward (WR) to boost training. Our proposedmethods (SG+RWS+WR) achieve BLEU-4 / CIDEr / SPICEscores of 36.6 / 116.9 / 21.3 with cross-entropy loss and 38.7 /129.1 / 22.4 with CIDEr optimization on MSCOCO Karpathytest split, which surpass the results of other state-of-the-artmethods.
Submission history
From: Li Wang [view email][v1] Wed, 15 Jan 2020 16:32:51 GMT (1377kb,D)
[v2] Fri, 11 Sep 2020 13:09:06 GMT (696kb,D)
[v3] Fri, 12 Mar 2021 05:00:56 GMT (696kb,D)
Link back to: arXiv, form interface, contact.