Current browse context:
cs.CL
Change to browse by:
References & Citations
Computer Science > Computation and Language
Title: Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech
(Submitted on 21 Nov 2019 (v1), last revised 14 Feb 2020 (this version, v2))
Abstract: In this paper, we present a method for learning discrete linguistic units by incorporating vector quantization layers into neural models of visually grounded speech. We show that our method is capable of capturing both word-level and sub-word units, depending on how it is configured. What differentiates this paper from prior work on speech unit learning is the choice of training objective. Rather than using a reconstruction-based loss, we use a discriminative, multimodal grounding objective which forces the learned units to be useful for semantic image retrieval. We evaluate the sub-word units on the ZeroSpeech 2019 challenge, achieving a 27.3\% reduction in ABX error rate over the top-performing submission, while keeping the bitrate approximately the same. We also present experiments demonstrating the noise robustness of these units. Finally, we show that a model with multiple quantizers can simultaneously learn phone-like detectors at a lower layer and word-like detectors at a higher layer. We show that these detectors are highly accurate, discovering 279 words with an F1 score of greater than 0.5.
Submission history
From: David Harwath [view email][v1] Thu, 21 Nov 2019 16:54:14 GMT (565kb,D)
[v2] Fri, 14 Feb 2020 18:38:28 GMT (1918kb,D)
Link back to: arXiv, form interface, contact.