Current browse context:
eess.AS
Change to browse by:
References & Citations
Electrical Engineering and Systems Science > Audio and Speech Processing
Title: Design Guidelines for Inclusive Speaker Verification Evaluation Datasets
(Submitted on 5 Apr 2022 (v1), last revised 13 Sep 2022 (this version, v2))
Abstract: Speaker verification (SV) provides billions of voice-enabled devices with access control, and ensures the security of voice-driven technologies. As a type of biometrics, it is necessary that SV is unbiased, with consistent and reliable performance across speakers irrespective of their demographic, social and economic attributes. Current SV evaluation practices are insufficient for evaluating bias: they are over-simplified and aggregate users, not representative of real-life usage scenarios, and consequences of errors are not accounted for. This paper proposes design guidelines for constructing SV evaluation datasets that address these short-comings. We propose a schema for grading the difficulty of utterance pairs, and present an algorithm for generating inclusive SV datasets. We empirically validate our proposed method in a set of experiments on the VoxCeleb1 dataset. Our results confirm that the count of utterance pairs/speaker, and the difficulty grading of utterance pairs have a significant effect on evaluation performance and variability. Our work contributes to the development of SV evaluation practices that are inclusive and fair.
Submission history
From: Wiebke Toussaint Hutiri [view email][v1] Tue, 5 Apr 2022 15:28:26 GMT (253kb,D)
[v2] Tue, 13 Sep 2022 13:05:52 GMT (276kb,D)
Link back to: arXiv, form interface, contact.