Current browse context:
cs.CL
Change to browse by:
References & Citations
Computer Science > Computation and Language
Title: How to Estimate Model Transferability of Pre-Trained Speech Models?
(Submitted on 1 Jun 2023 (v1), last revised 6 Feb 2024 (this version, v3))
Abstract: In this work, we introduce a "score-based assessment" framework for estimating the transferability of pre-trained speech models (PSMs) for fine-tuning target tasks. We leverage upon two representation theories, Bayesian likelihood estimation and optimal transport, to generate rank scores for the PSM candidates using the extracted representations. Our framework efficiently computes transferability scores without actual fine-tuning of candidate models or layers by making a temporal independent hypothesis. We evaluate some popular supervised speech models (e.g., Conformer RNN-Transducer) and self-supervised speech models (e.g., HuBERT) in cross-layer and cross-model settings using public data. Experimental results show a high Spearman's rank correlation and low $p$-value between our estimation framework and fine-tuning ground truth. Our proposed transferability framework requires less computational time and resources, making it a resource-saving and time-efficient approach for tuning speech foundation models.
Submission history
From: Huck Yang [view email][v1] Thu, 1 Jun 2023 04:52:26 GMT (1744kb,D)
[v2] Fri, 25 Aug 2023 13:50:19 GMT (1744kb,D)
[v3] Tue, 6 Feb 2024 03:52:48 GMT (1744kb,D)
Link back to: arXiv, form interface, contact.