Audio and Speech Processing
New submissions
[ showing up to 1000 entries per page: fewer | more ]
New submissions for Wed, 24 Apr 24
- [1] arXiv:2404.14564 [pdf, other]
-
Title: Exploring the Potential of Data-Driven Spatial Audio Enhancement Using a Single-Channel ModelSubjects: Audio and Speech Processing (eess.AS); Sound (cs.SD)
One key aspect differentiating data-driven single- and multi-channel speech enhancement and dereverberation methods is that both the problem formulation and complexity of the solutions are considerably more challenging in the latter case. Additionally, with limited computational resources, it is cumbersome to train models that require the management of larger datasets or those with more complex designs. In this scenario, an unverified hypothesis that single-channel methods can be adapted to multi-channel scenarios simply by processing each channel independently holds significant implications, boosting compatibility between sound scene capture and system input-output formats, while also allowing modern research to focus on other challenging aspects, such as full-bandwidth audio enhancement, competitive noise suppression, and unsupervised learning. This study verifies this hypothesis by comparing the enhancement promoted by a basic single-channel speech enhancement and dereverberation model with two other multi-channel models tailored to separate clean speech from noisy 3D mixes. A direction of arrival estimation model was used to objectively evaluate its capacity to preserve spatial information by comparing the output signals with ground-truth coordinate values. Consequently, a trade-off arises between preserving spatial information with a more straightforward single-channel solution at the cost of obtaining lower gains in intelligibility scores.
- [2] arXiv:2404.14700 [pdf, other]
-
Title: FlashSpeech: Efficient Zero-Shot Speech SynthesisAuthors: Zhen Ye, Zeqian Ju, Haohe Liu, Xu Tan, Jianyi Chen, Yiwen Lu, Peiwen Sun, Jiahao Pan, Weizhen Bian, Shulin He, Qifeng Liu, Yike Guo, Wei XueComments: Efficient zero-shot speech synthesisSubjects: Audio and Speech Processing (eess.AS); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG); Sound (cs.SD)
Recent progress in large-scale zero-shot speech synthesis has been significantly advanced by language models and diffusion models. However, the generation process of both methods is slow and computationally intensive. Efficient speech synthesis using a lower computing budget to achieve quality on par with previous work remains a significant challenge. In this paper, we present FlashSpeech, a large-scale zero-shot speech synthesis system with approximately 5\% of the inference time compared with previous work. FlashSpeech is built on the latent consistency model and applies a novel adversarial consistency training approach that can train from scratch without the need for a pre-trained diffusion model as the teacher. Furthermore, a new prosody generator module enhances the diversity of prosody, making the rhythm of the speech sound more natural. The generation processes of FlashSpeech can be achieved efficiently with one or two sampling steps while maintaining high audio quality and high similarity to the audio prompt for zero-shot speech generation. Our experimental results demonstrate the superior performance of FlashSpeech. Notably, FlashSpeech can be about 20 times faster than other zero-shot speech synthesis systems while maintaining comparable performance in terms of voice quality and similarity. Furthermore, FlashSpeech demonstrates its versatility by efficiently performing tasks like voice conversion, speech editing, and diverse speech sampling. Audio samples can be found in https://flashspeech.github.io/.
- [3] arXiv:2404.14860 [pdf, other]
-
Title: Rethinking Processing Distortions: Disentangling the Impact of Speech Enhancement Errors on Speech Recognition PerformanceAuthors: Tsubasa Ochiai, Kazuma Iwamoto, Marc Delcroix, Rintaro Ikeshita, Hiroshi Sato, Shoko Araki, Shigeru KatagiriComments: 13 pages, 6 figures, Submitted to IEEE/ACM Trans. Audio, Speech, and Language ProcessingSubjects: Audio and Speech Processing (eess.AS); Sound (cs.SD)
It is challenging to improve automatic speech recognition (ASR) performance in noisy conditions with a single-channel speech enhancement (SE) front-end. This is generally attributed to the processing distortions caused by the nonlinear processing of single-channel SE front-ends. However, the causes of such degraded ASR performance have not been fully investigated. How to design single-channel SE front-ends in a way that significantly improves ASR performance remains an open research question. In this study, we investigate a signal-level numerical metric that can explain the cause of degradation in ASR performance. To this end, we propose a novel analysis scheme based on the orthogonal projection-based decomposition of SE errors. This scheme manually modifies the ratio of the decomposed interference, noise, and artifact errors, and it enables us to directly evaluate the impact of each error type on ASR performance. Our analysis reveals the particularly detrimental effect of artifact errors on ASR performance compared to the other types of errors. This provides us with a more principled definition of processing distortions that cause the ASR performance degradation. Then, we study two practical approaches for reducing the impact of artifact errors. First, we prove that the simple observation adding (OA) post-processing (i.e., interpolating the enhanced and observed signals) can monotonically improve the signal-to-artifact ratio. Second, we propose a novel training objective, called artifact-boosted signal-to-distortion ratio (AB-SDR), which forces the model to estimate the enhanced signals with fewer artifact errors. Through experiments, we confirm that both the OA and AB-SDR approaches are effective in decreasing artifact errors caused by single-channel SE front-ends, allowing them to significantly improve ASR performance.
- [4] arXiv:2404.14903 [pdf, other]
-
Title: Multi-Sample Dynamic Time Warping for Few-Shot Keyword SpottingSubjects: Audio and Speech Processing (eess.AS); Information Retrieval (cs.IR); Sound (cs.SD)
In multi-sample keyword spotting, each keyword class is represented by multiple spoken instances, called samples. A na\"ive approach to detect keywords in a target sequence consists of querying all samples of all classes using sub-sequence dynamic time warping. However, the resulting processing time increases linearly with respect to the number of samples belonging to each class. Alternatively, only a single Fr\'echet mean can be queried for each class, resulting in reduced processing time but usually also in worse detection performance as the variability of the query samples is not captured sufficiently well. In this work, multi-sample dynamic time warping is proposed to compute class-specific cost-tensors that include the variability of all query samples. To significantly reduce the computational complexity during inference, these cost tensors are converted to cost matrices before applying dynamic time warping. In experimental evaluations for few-shot keyword spotting, it is shown that this method yields a very similar performance as using all individual query samples as templates while having a runtime that is only slightly slower than when using Fr\'echet means.
- [5] arXiv:2404.14913 [pdf, other]
-
Title: Additive Margin in Contrastive Self-Supervised Frameworks to Learn Discriminative Speaker RepresentationsComments: accepted at Odyssey 2024: The Speaker and Language Recognition Workshop. arXiv admin note: text overlap with arXiv:2306.03664Subjects: Audio and Speech Processing (eess.AS); Machine Learning (cs.LG); Sound (cs.SD)
Self-Supervised Learning (SSL) frameworks became the standard for learning robust class representations by benefiting from large unlabeled datasets. For Speaker Verification (SV), most SSL systems rely on contrastive-based loss functions. We explore different ways to improve the performance of these techniques by revisiting the NT-Xent contrastive loss. Our main contribution is the definition of the NT-Xent-AM loss and the study of the importance of Additive Margin (AM) in SimCLR and MoCo SSL methods to further separate positive from negative pairs. Despite class collisions, we show that AM enhances the compactness of same-speaker embeddings and reduces the number of false negatives and false positives on SV. Additionally, we demonstrate the effectiveness of the symmetric contrastive loss, which provides more supervision for the SSL task. Implementing these two modifications to SimCLR improves performance and results in 7.85% EER on VoxCeleb1-O, outperforming other equivalent methods.
- [6] arXiv:2404.15168 [pdf, other]
-
Title: Artificial Neural Networks to Recognize Speakers Division from Continuous Bengali SpeechSubjects: Audio and Speech Processing (eess.AS); Human-Computer Interaction (cs.HC); Machine Learning (cs.LG); Sound (cs.SD)
Voice based applications are ruling over the era of automation because speech has a lot of factors that determine a speakers information as well as speech. Modern Automatic Speech Recognition (ASR) is a blessing in the field of Human-Computer Interaction (HCI) for efficient communication among humans and devices using Artificial Intelligence technology. Speech is one of the easiest mediums of communication because it has a lot of identical features for different speakers. Nowadays it is possible to determine speakers and their identity using their speech in terms of speaker recognition. In this paper, we presented a method that will provide a speakers geographical identity in a certain region using continuous Bengali speech. We consider eight different divisions of Bangladesh as the geographical region. We applied the Mel Frequency Cepstral Coefficient (MFCC) and Delta features on an Artificial Neural Network to classify speakers division. We performed some preprocessing tasks like noise reduction and 8-10 second segmentation of raw audio before feature extraction. We used our dataset of more than 45 hours of audio data from 633 individual male and female speakers. We recorded the highest accuracy of 85.44%.
- [7] arXiv:2404.15176 [pdf, other]
-
Title: Voice Passing : a Non-Binary Voice Gender Prediction System for evaluating Transgender voice transitionAuthors: David Doukhan, Simon Devauchelle, Lucile Girard-Monneron, Mía Chávez Ruz, V. Chaddouk, Isabelle Wagner, Albert RilliardComments: 5 pages, 1 figure, keywords: Transgender voice, Gender perception, Speaker gender classification, CNN, X-VectorJournal-ref: Proc. INTERSPEECH 2023, 5207-5211Subjects: Audio and Speech Processing (eess.AS); Human-Computer Interaction (cs.HC); Machine Learning (cs.LG); Sound (cs.SD)
This paper presents a software allowing to describe voices using a continuous Voice Femininity Percentage (VFP). This system is intended for transgender speakers during their voice transition and for voice therapists supporting them in this process. A corpus of 41 French cis- and transgender speakers was recorded. A perceptual evaluation allowed 57 participants to estimate the VFP for each voice. Binary gender classification models were trained on external gender-balanced data and used on overlapping windows to obtain average gender prediction estimates, which were calibrated to predict VFP and obtained higher accuracy than $F_0$ or vocal track length-based models. Training data speaking style and DNN architecture were shown to impact VFP estimation. Accuracy of the models was affected by speakers' age. This highlights the importance of style, age, and the conception of gender as binary or not, to build adequate statistical representations of cultural concepts.
Cross-lists for Wed, 24 Apr 24
- [8] arXiv:2404.14716 (cross-list from cs.CL) [pdf, other]
-
Title: Bayesian Example Selection Improves In-Context Learning for Speech, Text, and Visual ModalitiesComments: 16 pages, 6 figuresSubjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Sound (cs.SD); Audio and Speech Processing (eess.AS)
Large language models (LLMs) can adapt to new tasks through in-context learning (ICL) based on a few examples presented in dialogue history without any model parameter update. Despite such convenience, the performance of ICL heavily depends on the quality of the in-context examples presented, which makes the in-context example selection approach a critical choice. This paper proposes a novel Bayesian in-Context example Selection method (ByCS) for ICL. Extending the inference probability conditioned on in-context examples based on Bayes' theorem, ByCS focuses on the inverse inference conditioned on test input. Following the assumption that accurate inverse inference probability (likelihood) will result in accurate inference probability (posterior), in-context examples are selected based on their inverse inference results. Diverse and extensive cross-tasking and cross-modality experiments are performed with speech, text, and image examples. Experimental results show the efficacy and robustness of our ByCS method on various models, tasks and modalities.
- [9] arXiv:2404.14736 (cross-list from cs.HC) [pdf, ps, other]
-
Title: Qualitative Approaches to Voice UXJournal-ref: ACM Computing Surveys (2024)Subjects: Human-Computer Interaction (cs.HC); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computers and Society (cs.CY); Sound (cs.SD); Audio and Speech Processing (eess.AS)
Voice is a natural mode of expression offered by modern computer-based systems. Qualitative perspectives on voice-based user experiences (voice UX) offer rich descriptions of complex interactions that numbers alone cannot fully represent. We conducted a systematic review of the literature on qualitative approaches to voice UX, capturing the nature of this body of work in a systematic map and offering a qualitative synthesis of findings. We highlight the benefits of qualitative methods for voice UX research, identify opportunities for increasing rigour in methods and outcomes, and distill patterns of experience across a diversity of devices and modes of qualitative praxis.
- [10] arXiv:2404.14946 (cross-list from cs.SD) [pdf, other]
-
Title: StoryTTS: A Highly Expressive Text-to-Speech Dataset with Rich Textual Expressiveness AnnotationsComments: Accepted by ICASSP 2024Journal-ref: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 11521-11525Subjects: Sound (cs.SD); Computation and Language (cs.CL); Audio and Speech Processing (eess.AS)
While acoustic expressiveness has long been studied in expressive text-to-speech (ETTS), the inherent expressiveness in text lacks sufficient attention, especially for ETTS of artistic works. In this paper, we introduce StoryTTS, a highly ETTS dataset that contains rich expressiveness both in acoustic and textual perspective, from the recording of a Mandarin storytelling show. A systematic and comprehensive labeling framework is proposed for textual expressiveness. We analyze and define speech-related textual expressiveness in StoryTTS to include five distinct dimensions through linguistics, rhetoric, etc. Then we employ large language models and prompt them with a few manual annotation examples for batch annotation. The resulting corpus contains 61 hours of consecutive and highly prosodic speech equipped with accurate text transcriptions and rich textual expressiveness annotations. Therefore, StoryTTS can aid future ETTS research to fully mine the abundant intrinsic textual and acoustic features. Experiments are conducted to validate that TTS models can generate speech with improved expressiveness when integrating with the annotated textual labels in StoryTTS.
- [11] arXiv:2404.15143 (cross-list from cs.SD) [pdf, other]
-
Title: Every Breath You Don't Take: Deepfake Speech Detection Using BreathAuthors: Seth Layton, Thiago De Andrade, Daniel Olszewski, Kevin Warren, Carrie Gates, Kevin Butler, Patrick TraynorComments: Submitted to ACM journal -- Digital Threats: Research and PracticeSubjects: Sound (cs.SD); Cryptography and Security (cs.CR); Multimedia (cs.MM); Audio and Speech Processing (eess.AS)
Deepfake speech represents a real and growing threat to systems and society. Many detectors have been created to aid in defense against speech deepfakes. While these detectors implement myriad methodologies, many rely on low-level fragments of the speech generation process. We hypothesize that breath, a higher-level part of speech, is a key component of natural speech and thus improper generation in deepfake speech is a performant discriminator. To evaluate this, we create a breath detector and leverage this against a custom dataset of online news article audio to discriminate between real/deepfake speech. Additionally, we make this custom dataset publicly available to facilitate comparison for future work. Applying our simple breath detector as a deepfake speech discriminator on in-the-wild samples allows for accurate classification (perfect 1.0 AUPRC and 0.0 EER on test data) across 33.6 hours of audio. We compare our model with the state-of-the-art SSL-wav2vec model and show that this complex deep learning model completely fails to classify the same in-the-wild samples (0.72 AUPRC and 0.99 EER).
- [12] arXiv:2404.15181 (cross-list from cs.SD) [pdf, ps, other]
-
Title: Tailors: New Music Timbre Visualizer to Entertain Music Through ImageryAuthors: ChungHa LeeComments: 47 pages, 9 figures, 5 tablesSubjects: Sound (cs.SD); Human-Computer Interaction (cs.HC); Audio and Speech Processing (eess.AS)
In this paper, I have implemented a timbre visualization system called Tailors. Through the experiment with 27 MIR users, Tailors was found to be effective in conveying timbral warmth, brightness, depth, shallowness, hardness, roughness, and sharpness features of music compared to the only music condition and basic visualization. All scores of Tailors in the music imagery and music entertainment surveys were valued highest among the three conditions. Multiple linear regression analysis between timbre-imagery and imagery-entertainment shows significant and positive correlations. Coefficients comparing results from Fisher Transformation show that Tailors made user's music entertainment better through improved music visual imagery. The post-survey result represents that Tailors ranked first for the best timbre expression, music experience, and willingness to use it again. While some users felt a burden in the eye, Tailors left the future work of the data-driven approach of the mapping rule of timbre visualization to gain consent from many users. Furthermore, reducing timbre features to focus on features that Tailors can express well was also discussed, with future work of Tailors in a more artistic way using the sense of space.
- [13] arXiv:2404.15208 (cross-list from cs.SI) [pdf, other]
-
Title: Analysis and Visualization of Musical Structure using NetworksSubjects: Social and Information Networks (cs.SI); Sound (cs.SD); Audio and Speech Processing (eess.AS); Physics and Society (physics.soc-ph)
In this article, a framework for defining and analysing a family of graphs or networks from symbolic music information is discussed. Such graphs concern different types of elements, such as pitches, chords and rhythms, and the relations among them, and are built from quantitative or categorical data contained in digital music scores. They are helpful in visualizing musical features at once, thus leading to a computational tool for understanding the general structural elements of a music fragment. Data obtained from a digital score undergoes different analytical procedures from graph and network theory, such as computing their centrality measures and entropy, and detecting their communities. We analyze pieces of music coming from different styles, and compare some of our results with conclusions from traditional music analysis techniques.
Replacements for Wed, 24 Apr 24
- [14] arXiv:2310.15845 (replaced) [pdf, other]
-
Title: Pre-training Music Classification Models via Music Source SeparationComments: 5 pages, 2 figures/2 tables. EUSIPCO-24 submission (changes from v1: experiments with AST - classification backbone pre-training - correct FMA subset)Subjects: Audio and Speech Processing (eess.AS)
- [15] arXiv:2403.03100 (replaced) [pdf, other]
-
Title: NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion ModelsAuthors: Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, Zhizheng Wu, Tao Qin, Xiang-Yang Li, Wei Ye, Shikun Zhang, Jiang Bian, Lei He, Jinyu Li, Sheng ZhaoComments: Achieving human-level quality and naturalness on multi-speaker datasets (e.g., LibriSpeech) in a zero-shot waySubjects: Audio and Speech Processing (eess.AS); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG); Sound (cs.SD)
- [16] arXiv:2403.09524 (replaced) [pdf, other]
-
Title: Physics-Informed Neural Network for Volumetric Sound field Reconstruction of Speech SignalsAuthors: Marco Olivieri, Xenofon Karakonstantis, Mirco Pezzoli, Fabio Antonacci, Augusto Sarti, Efren Fernandez-GrandeSubjects: Audio and Speech Processing (eess.AS)
- [17] arXiv:2306.00107 (replaced) [pdf, other]
-
Title: MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised TrainingAuthors: Yizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma, Xingran Chen, Hanzhi Yin, Chenghao Xiao, Chenghua Lin, Anton Ragni, Emmanouil Benetos, Norbert Gyenge, Roger Dannenberg, Ruibo Liu, Wenhu Chen, Gus Xia, Yemin Shi, Wenhao Huang, Zili Wang, Yike Guo, Jie FuComments: accepted by ICLR 2024Subjects: Sound (cs.SD); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG); Audio and Speech Processing (eess.AS)
- [18] arXiv:2312.17156 (replaced) [pdf, other]
-
Title: BEAST: Online Joint Beat and Downbeat Tracking Based on Streaming TransformerComments: Accepted by ICASSP 2024Subjects: Sound (cs.SD); Audio and Speech Processing (eess.AS)
- [19] arXiv:2404.10180 (replaced) [pdf, other]
-
Title: Deferred NAM: Low-latency Top-K Context Injection via Deferred Context Encoding for Non-Streaming ASRAuthors: Zelin Wu, Gan Song, Christopher Li, Pat Rondon, Zhong Meng, Xavier Velez, Weiran Wang, Diamantino Caseiro, Golan Pundak, Tsendsuren Munkhdalai, Angad Chandorkar, Rohit PrabhavalkarComments: 9 pages, 3 figures, accepted by NAACL 2024 - Industry TrackJournal-ref: 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics - Industry TrackSubjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE); Audio and Speech Processing (eess.AS)
- [20] arXiv:2404.13569 (replaced) [pdf, other]
-
Title: Musical Word Embedding for Music Tagging and RetrievalComments: Submitted to IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP)Subjects: Sound (cs.SD); Audio and Speech Processing (eess.AS)
- [21] arXiv:2404.13892 (replaced) [pdf, other]
-
Title: Retrieval-Augmented Audio Deepfake DetectionComments: Accepted by the 2024 International Conference on Multimedia Retrieval (ICMR 2024)Subjects: Sound (cs.SD); Artificial Intelligence (cs.AI); Audio and Speech Processing (eess.AS)
[ showing up to 1000 entries per page: fewer | more ]
Disable MathJax (What is MathJax?)
Links to: arXiv, form interface, find, eess, recent, 2404, contact, help (Access key information)