We gratefully acknowledge support from
the Simons Foundation and member institutions.

Audio and Speech Processing

New submissions

[ total of 19 entries: 1-19 ]
[ showing up to 2000 entries per page: fewer | more ]

New submissions for Wed, 5 Oct 22

[1]  arXiv:2210.01273 [pdf, other]
Title: An attention-based backend allowing efficient fine-tuning of transformer models for speaker verification
Comments: Accepted by SLT2022
Subjects: Audio and Speech Processing (eess.AS)

In recent years, self-supervised learning paradigm has received extensive attention due to its great success in various down-stream tasks. However, the fine-tuning strategies for adapting those pre-trained models to speaker verification task have yet to be fully explored. In this paper, we analyze several feature extraction approaches built on top of a pre-trained model, as well as regularization and learning rate schedule to stabilize the fine-tuning process and further boost performance: multi-head factorized attentive pooling is proposed to factorize the comparison of speaker representations into multiple phonetic clusters. We regularize towards the parameters of the pre-trained model and we set different learning rates for each layer of the pre-trained model during fine-tuning. The experimental results show our method can significantly shorten the training time to 4 hours and achieve SOTA performance: 0.59%, 0.79% and 1.77% EER on Vox1-O, Vox1-E and Vox1-H, respectively.

[2]  arXiv:2210.01677 [pdf, ps, other]
Title: The DKU-DukeECE Diarization System for the VoxCeleb Speaker Recognition Challenge 2022
Comments: arXiv admin note: substantial text overlap with arXiv:2109.02002
Subjects: Audio and Speech Processing (eess.AS)

This paper discribes the DKU-DukeECE submission to the 4th track of the VoxCeleb Speaker Recognition Challenge 2022 (VoxSRC-22). Our system contains a fused voice activity detection model, a clustering-based diarization model, and a target-speaker voice activity detection-based overlap detection model. Overall, the submitted system is similar to our previous year's system in VoxSRC-21. The difference is that we use a much better speaker embedding and a fused voice activity detection, which significantly improves the performance. Finally, we fuse 4 different systems using DOVER-lap and achieve 4.75 of the diarization error rate, which ranks the 1st place in track 4.

Cross-lists for Wed, 5 Oct 22

[3]  arXiv:2210.01205 (cross-list from cs.LG) [pdf]
Title: Diagnosis of Parkinson's Disease Based on Voice Signals Using SHAP and Hard Voting Ensemble Method
Subjects: Machine Learning (cs.LG); Audio and Speech Processing (eess.AS); Signal Processing (eess.SP)

Background and Objective: Parkinson's disease (PD) is the second most common progressive neurological condition after Alzheimer's, characterized by motor and non-motor symptoms. Developing a method to diagnose the condition in its beginning phases is essential because of the significant number of individuals afflicting with this illness. PD is typically identified using motor symptoms or other Neuroimaging techniques, such as DATSCAN and SPECT. These methods are expensive, time-consuming, and unavailable to the general public; furthermore, they are not very accurate. These constraints encouraged us to develop a novel technique using SHAP and Hard Voting Ensemble Method based on voice signals. Methods: In this article, we used Pearson Correlation Coefficients to understand the relationship between input features and the output, and finally, input features with high correlation were selected. These selected features were classified by the Extreme Gradient Boosting (XGBoost), Light Gradient Boosting Machine (LightGBM), Gradient Boosting, and Bagging. Moreover, the Hard Voting Ensemble Method was determined based on the performance of the four classifiers. At the final stage, we proposed Shapley Additive exPlanations (SHAP) to rank the features according to their significance in diagnosing Parkinson's disease. Results and Conclusion: The proposed method achieved 85.42% accuracy, 84.94% F1-score, 86.77% precision, 87.62% specificity, and 83.20% sensitivity. The study's findings demonstrated that the proposed method outperformed state-of-the-art approaches and can assist physicians in diagnosing Parkinson's cases.

[4]  arXiv:2210.01256 (cross-list from cs.SD) [pdf, ps, other]
Title: And what if two musical versions don't share melody, harmony, rhythm, or lyrics ?
Subjects: Sound (cs.SD); Machine Learning (cs.LG); Audio and Speech Processing (eess.AS)

Version identification (VI) has seen substantial progress over the past few years. On the one hand, the introduction of the metric learning paradigm has favored the emergence of scalable yet accurate VI systems. On the other hand, using features focusing on specific aspects of musical pieces, such as melody, harmony, or lyrics, yielded interpretable and promising performances. In this work, we build upon these recent advances and propose a metric learning-based system systematically leveraging four dimensions commonly admitted to convey musical similarity between versions: melodic line, harmonic structure, rhythmic patterns, and lyrics. We describe our deliberately simple model architecture, and we show in particular that an approximated representation of the lyrics is an efficient proxy to discriminate between versions and non-versions. We then describe how these features complement each other and yield new state-of-the-art performances on two publicly available datasets. We finally suggest that a VI system using a combination of melodic, harmonic, rhythmic and lyrics features could theoretically reach the optimal performances obtainable on these datasets.

[5]  arXiv:2210.01353 (cross-list from cs.SD) [pdf, other]
Title: Pay Self-Attention to Audio-Visual Navigation
Subjects: Sound (cs.SD); Artificial Intelligence (cs.AI); Audio and Speech Processing (eess.AS)

Audio-visual embodied navigation, as a hot research topic, aims training a robot to reach an audio target using egocentric visual (from the sensors mounted on the robot) and audio (emitted from the target) input. The audio-visual information fusion strategy is naturally important to the navigation performance, but the state-of-the-art methods still simply concatenate the visual and audio features, potentially ignoring the direct impact of context. Moreover, the existing approaches requires either phase-wise training or additional aid (e.g. topology graph and sound semantics). Up till this date, the work that deals with the more challenging setup with moving target(s) is still rare. As a result, we propose an end-to-end framework FSAAVN (feature self-attention audio-visual navigation) to learn chasing after a moving audio target using a context-aware audio-visual fusion strategy implemented as a self-attention module. Our thorough experiments validate the superior performance (both quantitatively and qualitatively) of FSAAVN in comparison with the state-of-the-arts, and also provide unique insights about the choice of visual modalities, visual/audio encoder backbones and fusion patterns.

[6]  arXiv:2210.01448 (cross-list from cs.SD) [pdf, other]
Title: Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddings
Comments: SIGGRAPH Asia 2022 (Journal Track); Project Page: this https URL
Subjects: Sound (cs.SD); Artificial Intelligence (cs.AI); Graphics (cs.GR); Audio and Speech Processing (eess.AS)

Automatic synthesis of realistic co-speech gestures is an increasingly important yet challenging task in artificial embodied agent creation. Previous systems mainly focus on generating gestures in an end-to-end manner, which leads to difficulties in mining the clear rhythm and semantics due to the complex yet subtle harmony between speech and gestures. We present a novel co-speech gesture synthesis method that achieves convincing results both on the rhythm and semantics. For the rhythm, our system contains a robust rhythm-based segmentation pipeline to ensure the temporal coherence between the vocalization and gestures explicitly. For the gesture semantics, we devise a mechanism to effectively disentangle both low- and high-level neural embeddings of speech and motion based on linguistic theory. The high-level embedding corresponds to semantics, while the low-level embedding relates to subtle variations. Lastly, we build correspondence between the hierarchical embeddings of the speech and the motion, resulting in rhythm- and semantics-aware gesture synthesis. Evaluations with existing objective metrics, a newly proposed rhythmic metric, and human feedback show that our method outperforms state-of-the-art systems by a clear margin.

[7]  arXiv:2210.01512 (cross-list from cs.CL) [pdf, other]
Title: Code-Switching without Switching: Language Agnostic End-to-End Speech Translation
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG); Sound (cs.SD); Audio and Speech Processing (eess.AS)

We propose a) a Language Agnostic end-to-end Speech Translation model (LAST), and b) a data augmentation strategy to increase code-switching (CS) performance. With increasing globalization, multiple languages are increasingly used interchangeably during fluent speech. Such CS complicates traditional speech recognition and translation, as we must recognize which language was spoken first and then apply a language-dependent recognizer and subsequent translation component to generate the desired target language output. Such a pipeline introduces latency and errors. In this paper, we eliminate the need for that, by treating speech recognition and translation as one unified end-to-end speech translation problem. By training LAST with both input languages, we decode speech into one target language, regardless of the input language. LAST delivers comparable recognition and speech translation accuracy in monolingual usage, while reducing latency and error rate considerably when CS is observed.

[8]  arXiv:2210.01703 (cross-list from cs.SD) [pdf, other]
Title: Improving Label-Deficient Keyword Spotting Using Self-Supervised Pretraining
Authors: Holger Severin Bovbjerg (1), Zheng-Hua Tan (1) ((1) Aalborg University)
Comments: 8 pages, 3 figures, 4 tables, Submitted to Northern Lights Deep Learning Conference 2023
Subjects: Sound (cs.SD); Human-Computer Interaction (cs.HC); Machine Learning (cs.LG); Audio and Speech Processing (eess.AS)

In recent years, the development of accurate deep keyword spotting (KWS) models has resulted in KWS technology being embedded in a number of technologies such as voice assistants. Many of these models rely on large amounts of labelled data to achieve good performance. As a result, their use is restricted to applications for which a large labelled speech data set can be obtained. Self-supervised learning seeks to mitigate the need for large labelled data sets by leveraging unlabelled data, which is easier to obtain in large amounts. However, most self-supervised methods have only been investigated for very large models, whereas KWS models are desired to be small. In this paper, we investigate the use of self-supervised pretraining for the smaller KWS models in a label-deficient scenario. We pretrain the Keyword Transformer model using the self-supervised framework Data2Vec and carry out experiments on a label-deficient setup of the Google Speech Commands data set. It is found that the pretrained models greatly outperform the models without pretraining, showing that Data2Vec pretraining can increase the performance of KWS models in label-deficient scenarios. The source code is made publicly available.

[9]  arXiv:2210.01719 (cross-list from cs.SD) [pdf, other]
Title: Learning the Spectrogram Temporal Resolution for Audio Classification
Comments: Under review. Code open-sourced at this https URL
Subjects: Sound (cs.SD); Artificial Intelligence (cs.AI); Multimedia (cs.MM); Audio and Speech Processing (eess.AS); Signal Processing (eess.SP)

The audio spectrogram is a time-frequency representation that has been widely used for audio classification. The temporal resolution of a spectrogram depends on hop size. Previous works generally assume the hop size should be a constant value such as ten milliseconds. However, a fixed hop size or resolution is not always optimal for different types of sound. This paper proposes a novel method, DiffRes, that enables differentiable temporal resolution learning to improve the performance of audio classification models. Given a spectrogram calculated with a fixed hop size, DiffRes merges non-essential time frames while preserving important frames. DiffRes acts as a "drop-in" module between an audio spectrogram and a classifier, and can be end-to-end optimized. We evaluate DiffRes on the mel-spectrogram, followed by state-of-the-art classifier backbones, and apply it to five different subtasks. Compared with using the fixed-resolution mel-spectrogram, the DiffRes-based method can achieve the same or better classification accuracy with at least 25% fewer temporal dimensions on the feature level, which alleviates the computational cost at the same time. Starting from a high-temporal-resolution spectrogram such as one-millisecond hop size, we show that DiffRes can improve classification accuracy with the same computational complexity.

Replacements for Wed, 5 Oct 22

[10]  arXiv:2203.16222 (replaced) [pdf, other]
Title: Phase-Aware Deep Speech Enhancement: It's All About The Frame Length
Comments: The following article has been accepted by JASA Express Letters. After it is published, it will be found at this http URL
Subjects: Audio and Speech Processing (eess.AS); Machine Learning (cs.LG); Sound (cs.SD)
[11]  arXiv:2209.04974 (replaced) [pdf, other]
Title: VarArray Meets t-SOT: Advancing the State of the Art of Streaming Distant Conversational Speech Recognition
Comments: 6 pages, 2 figure, 3 tables, v2: Appendix A has been added
Subjects: Audio and Speech Processing (eess.AS); Computation and Language (cs.CL); Sound (cs.SD)
[12]  arXiv:2209.09967 (replaced) [src]
Title: Language-based Audio Retrieval Task in DCASE 2022 Challenge
Comments: Update for arXiv:2206.06108 mistakenly submitted as a new article
Subjects: Audio and Speech Processing (eess.AS); Sound (cs.SD)
[13]  arXiv:2107.12049 (replaced) [pdf, other]
Title: SVEva Fair: A Framework for Evaluating Fairness in Speaker Verification
Comments: This unpublished paper has been superseded by "Bias in Automated Speaker Recognition" (published at ACM FAccT 2022): arXiv:2201.09486
Subjects: Sound (cs.SD); Artificial Intelligence (cs.AI); Audio and Speech Processing (eess.AS)
[14]  arXiv:2111.02735 (replaced) [pdf, other]
Title: A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understanding
Comments: 7 pages, 2 figures
Subjects: Computation and Language (cs.CL); Neural and Evolutionary Computing (cs.NE); Sound (cs.SD); Audio and Speech Processing (eess.AS)
[15]  arXiv:2111.10639 (replaced) [pdf, other]
Title: Implicit Acoustic Echo Cancellation for Keyword Spotting and Device-Directed Speech Detection
Comments: To be presented at SLT 2022
Subjects: Sound (cs.SD); Machine Learning (cs.LG); Audio and Speech Processing (eess.AS)
[16]  arXiv:2204.09595 (replaced) [pdf, other]
Title: Exploring Continuous Integrate-and-Fire for Adaptive Simultaneous Speech Translation
Comments: INTERSPEECH 2022 camera ready
Journal-ref: Proc. Interspeech 2022, 5175-5179
Subjects: Computation and Language (cs.CL); Sound (cs.SD); Audio and Speech Processing (eess.AS)
[17]  arXiv:2208.03067 (replaced) [pdf, ps, other]
Title: Large vocabulary speech recognition for languages of Africa: multilingual modeling and self-supervised learning
Subjects: Computation and Language (cs.CL); Sound (cs.SD); Audio and Speech Processing (eess.AS)
[18]  arXiv:2208.11460 (replaced) [pdf, other]
Title: Improving Natural-Language-based Audio Retrieval with Transfer Learning and Audio & Text Augmentations
Comments: accepted at DCASE Workshop 2022
Subjects: Sound (cs.SD); Machine Learning (cs.LG); Audio and Speech Processing (eess.AS); Signal Processing (eess.SP)
[19]  arXiv:2209.15575 (replaced) [pdf, other]
Title: Match to Win: Analysing Sequences Lengths for Efficient Self-supervised Learning in Speech and Audio
Subjects: Sound (cs.SD); Machine Learning (cs.LG); Audio and Speech Processing (eess.AS)
[ total of 19 entries: 1-19 ]
[ showing up to 2000 entries per page: fewer | more ]

Disable MathJax (What is MathJax?)

Links to: arXiv, form interface, find, eess, recent, 2210, contact, help  (Access key information)