Current browse context:
cs.CL
Change to browse by:
References & Citations
Computer Science > Computation and Language
Title: "Listen, Understand and Translate": Triple Supervision Decouples End-to-end Speech-to-text Translation
(Submitted on 21 Sep 2020 (v1), last revised 5 Apr 2021 (this version, v3))
Abstract: An end-to-end speech-to-text translation (ST) takes audio in a source language and outputs the text in a target language. Existing methods are limited by the amount of parallel corpus. Can we build a system to fully utilize signals in a parallel ST corpus? We are inspired by human understanding system which is composed of auditory perception and cognitive processing. In this paper, we propose Listen-Understand-Translate, (LUT), a unified framework with triple supervision signals to decouple the end-to-end speech-to-text translation task. LUT is able to guide the acoustic encoder to extract as much information from the auditory input. In addition, LUT utilizes a pre-trained BERT model to enforce the upper encoder to produce as much semantic information as possible, without extra data. We perform experiments on a diverse set of speech translation benchmarks, including Librispeech English-French, IWSLT English-German and TED English-Chinese. Our results demonstrate LUT achieves the state-of-the-art performance, outperforming previous methods. The code is available at this https URL
Submission history
From: Qianqian Dong [view email][v1] Mon, 21 Sep 2020 09:19:07 GMT (7683kb,D)
[v2] Mon, 28 Dec 2020 07:28:44 GMT (671kb,D)
[v3] Mon, 5 Apr 2021 12:36:42 GMT (1296kb,D)
Link back to: arXiv, form interface, contact.