Current browse context:
eess.AS
Change to browse by:
References & Citations
Electrical Engineering and Systems Science > Audio and Speech Processing
Title: Word Discovery in Visually Grounded, Self-Supervised Speech Models
(Submitted on 28 Mar 2022 (v1), last revised 20 Jun 2023 (this version, v5))
Abstract: We present a method for visually-grounded spoken term discovery. After training either a HuBERT or wav2vec2.0 model to associate spoken captions with natural images, we show that powerful word segmentation and clustering capability emerges within the model's self-attention heads. Our experiments reveal that this ability is not present to nearly the same extent in the base HuBERT and wav2vec2.0 models, suggesting that the visual grounding task is a crucial component of the word discovery capability we observe. We also evaluate our method on the Buckeye word segmentation and ZeroSpeech spoken term discovery tasks, where we perform on par with or better than currently published methods on several metrics. Code and model weights are available at this https URL
Submission history
From: Puyuan Peng [view email][v1] Mon, 28 Mar 2022 20:41:17 GMT (1157kb,D)
[v2] Sun, 26 Jun 2022 01:31:26 GMT (2045kb,D)
[v3] Fri, 1 Jul 2022 16:58:08 GMT (2044kb,D)
[v4] Tue, 6 Sep 2022 15:36:06 GMT (1426kb,D)
[v5] Tue, 20 Jun 2023 01:55:28 GMT (1915kb,D)
Link back to: arXiv, form interface, contact.