Current browse context:
eess.AS
Change to browse by:
References & Citations
Electrical Engineering and Systems Science > Audio and Speech Processing
Title: MM-ALT: A Multimodal Automatic Lyric Transcription System
(Submitted on 13 Jul 2022 (v1), last revised 17 Feb 2023 (this version, v3))
Abstract: Automatic lyric transcription (ALT) is a nascent field of study attracting increasing interest from both the speech and music information retrieval communities, given its significant application potential. However, ALT with audio data alone is a notoriously difficult task due to instrumental accompaniment and musical constraints resulting in degradation of both the phonetic cues and the intelligibility of sung lyrics. To tackle this challenge, we propose the MultiModal Automatic Lyric Transcription system (MM-ALT), together with a new dataset, N20EM, which consists of audio recordings, videos of lip movements, and inertial measurement unit (IMU) data of an earbud worn by the performing singer. We first adapt the wav2vec 2.0 framework from automatic speech recognition (ASR) to the ALT task. We then propose a video-based ALT method and an IMU-based voice activity detection (VAD) method. In addition, we put forward the Residual Cross Attention (RCA) mechanism to fuse data from the three modalities (i.e., audio, video, and IMU). Experiments show the effectiveness of our proposed MM-ALT system, especially in terms of noise robustness. Project page is at this https URL
Submission history
From: Longshen Ou [view email][v1] Wed, 13 Jul 2022 11:23:34 GMT (2477kb,D)
[v2] Wed, 5 Oct 2022 06:15:31 GMT (6026kb,D)
[v3] Fri, 17 Feb 2023 12:21:02 GMT (6047kb,D)
Link back to: arXiv, form interface, contact.