References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Twitter-COMMs: Detecting Climate, COVID, and Military Multimodal Misinformation
(Submitted on 16 Dec 2021 (v1), last revised 3 May 2022 (this version, v2))
Abstract: Detecting out-of-context media, such as "mis-captioned" images on Twitter, is a relevant problem, especially in domains of high public significance. In this work we aim to develop defenses against such misinformation for the topics of Climate Change, COVID-19, and Military Vehicles. We first present a large-scale multimodal dataset with over 884k tweets relevant to these topics. Next, we propose a detection method, based on the state-of-the-art CLIP model, that leverages automatically generated hard image-text mismatches. While this approach works well on our automatically constructed out-of-context tweets, we aim to validate its usefulness on data representative of the real world. Thus, we test it on a set of human-generated fakes created by mimicking in-the-wild misinformation. We achieve an 11% detection improvement in a high precision regime over a strong baseline. Finally, we share insights about our best model design and analyze the challenges of this emerging threat.
Submission history
From: Giscard Biamby [view email][v1] Thu, 16 Dec 2021 03:37:20 GMT (5088kb,D)
[v2] Tue, 3 May 2022 00:51:02 GMT (4999kb,D)
Link back to: arXiv, form interface, contact.