We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.AI

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Artificial Intelligence

Title: ICAF: Iterative Contrastive Alignment Framework for Multimodal Abstractive Summarization

Abstract: Integrating multimodal knowledge for abstractive summarization task is a work-in-progress research area, with present techniques inheriting fusion-then-generation paradigm. Due to semantic gaps between computer vision and natural language processing, current methods often treat multiple data points as separate objects and rely on attention mechanisms to search for connection in order to fuse together. In addition, missing awareness of cross-modal matching from many frameworks leads to performance reduction. To solve these two drawbacks, we propose an Iterative Contrastive Alignment Framework (ICAF) that uses recurrent alignment and contrast to capture the coherences between images and texts. Specifically, we design a recurrent alignment (RA) layer to gradually investigate fine-grained semantical relationships between image patches and text tokens. At each step during the encoding process, cross-modal contrastive losses are applied to directly optimize the embedding space. According to ROUGE, relevance scores, and human evaluation, our model outperforms the state-of-the-art baselines on MSMO dataset. Experiments on the applicability of our proposed framework and hyperparameters settings have been also conducted.
Comments: Accepted by WCCI-IJCNN 2022 as an oral paper
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2108.05123 [cs.AI]
  (or arXiv:2108.05123v2 [cs.AI] for this version)

Submission history

From: Zijian Zhang [view email]
[v1] Wed, 11 Aug 2021 09:59:34 GMT (1593kb,D)
[v2] Wed, 27 Apr 2022 03:14:21 GMT (5600kb,D)

Link back to: arXiv, form interface, contact.