Current browse context:
eess.AS
Change to browse by:
References & Citations
Electrical Engineering and Systems Science > Audio and Speech Processing
Title: Unsupervised Audio-Caption Aligning Learns Correspondences between Individual Sound Events and Textual Phrases
(Submitted on 6 Oct 2021 (v1), last revised 21 Feb 2022 (this version, v2))
Abstract: We investigate unsupervised learning of correspondences between sound events and textual phrases through aligning audio clips with textual captions describing the content of a whole audio clip. We align originally unaligned and unannotated audio clips and their captions by scoring the similarities between audio frames and words, as encoded by modality-specific encoders and using a ranking-loss criterion to optimize the model. After training, we obtain clip-caption similarity by averaging frame-word similarities and estimate event-phrase correspondences by calculating frame-phrase similarities. We evaluate the method with two cross-modal tasks: audio-caption retrieval, and phrase-based sound event detection (SED). Experimental results show that the proposed method can globally associate audio clips with captions as well as locally learn correspondences between individual sound events and textual phrases in an unsupervised manner.
Submission history
From: Huang Xie [view email][v1] Wed, 6 Oct 2021 17:37:03 GMT (533kb,D)
[v2] Mon, 21 Feb 2022 06:27:11 GMT (534kb,D)
Link back to: arXiv, form interface, contact.