Current browse context:
cs.CL
Change to browse by:
References & Citations
Computer Science > Computation and Language
Title: On The Robustness of Self-Supervised Representations for Spoken Language Modeling
(Submitted on 30 Sep 2022 (this version), latest version 29 May 2023 (v2))
Abstract: Self-supervised representations have been extensively studied for discriminative and generative tasks. However, their robustness capabilities have not been extensively investigated. This work focuses on self-supervised representations for spoken generative language models. First, we empirically demonstrate how current state-of-the-art speech representation models lack robustness to basic signal variations that do not alter the spoken information. To overcome this, we propose an effective and efficient method to learn robust self-supervised speech representation for generative spoken language modeling. The proposed approach is based on applying a set of signal transformations to the speech signal and optimizing the model using an iterative pseudo-labeling scheme. Our method significantly improves over the evaluated baselines when considering encoding metrics. We additionally evaluate our method on the speech-to-speech translation task. We consider Spanish-English and French-English conversions and empirically demonstrate the benefits of following the proposed approach.
Submission history
From: Itai Gat [view email][v1] Fri, 30 Sep 2022 14:15:03 GMT (135kb,D)
[v2] Mon, 29 May 2023 10:50:29 GMT (123kb,D)
Link back to: arXiv, form interface, contact.