Current browse context:
cs.CL
Change to browse by:
References & Citations
Computer Science > Computation and Language
Title: Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM
(Submitted on 24 May 2023 (v1), last revised 20 Oct 2023 (this version, v3))
Abstract: We present a novel approach to adapting pre-trained large language models (LLMs) to perform question answering (QA) and speech continuation. By endowing the LLM with a pre-trained speech encoder, our model becomes able to take speech inputs and generate speech outputs. The entire system is trained end-to-end and operates directly on spectrograms, simplifying our architecture. Key to our approach is a training objective that jointly supervises speech recognition, text continuation, and speech synthesis using only paired speech-text pairs, enabling a `cross-modal' chain-of-thought within a single decoding pass. Our method surpasses existing spoken language models in speaker preservation and semantic coherence. Furthermore, the proposed model improves upon direct initialization in retaining the knowledge of the original LLM as demonstrated through spoken QA datasets. Audio samples can be found at this https URL
Submission history
From: Eliya Nachmani [view email][v1] Wed, 24 May 2023 15:39:43 GMT (184kb,D)
[v2] Thu, 1 Jun 2023 08:04:19 GMT (184kb,D)
[v3] Fri, 20 Oct 2023 05:55:39 GMT (225kb,D)
Link back to: arXiv, form interface, contact.