Current browse context:
cs.CV
Change to browse by:
References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Audio-Visual Person-of-Interest DeepFake Detection
(Submitted on 6 Apr 2022 (v1), last revised 18 May 2023 (this version, v3))
Abstract: Face manipulation technology is advancing very rapidly, and new methods are being proposed day by day. The aim of this work is to propose a deepfake detector that can cope with the wide variety of manipulation methods and scenarios encountered in the real world. Our key insight is that each person has specific characteristics that a synthetic generator likely cannot reproduce. Accordingly, we extract audio-visual features which characterize the identity of a person, and use them to create a person-of-interest (POI) deepfake detector. We leverage a contrastive learning paradigm to learn the moving-face and audio segment embeddings that are most discriminative for each identity. As a result, when the video and/or audio of a person is manipulated, its representation in the embedding space becomes inconsistent with the real identity, allowing reliable detection. Training is carried out exclusively on real talking-face video; thus, the detector does not depend on any specific manipulation method and yields the highest generalization ability. In addition, our method can detect both single-modality (audio-only, video-only) and multi-modality (audio-video) attacks, and is robust to low-quality or corrupted videos. Experiments on a wide variety of datasets confirm that our method ensures a SOTA performance, especially on low quality videos. Code is publicly available on-line at this https URL
Submission history
From: Davide Cozzolino [view email][v1] Wed, 6 Apr 2022 20:51:40 GMT (559kb,D)
[v2] Wed, 21 Dec 2022 23:07:27 GMT (547kb,D)
[v3] Thu, 18 May 2023 06:56:42 GMT (686kb,D)
Link back to: arXiv, form interface, contact.