Current browse context:
cs.CL
Change to browse by:
References & Citations
Computer Science > Computation and Language
Title: Where to start? Analyzing the potential value of intermediate models
(Submitted on 31 Oct 2022 (v1), last revised 10 Nov 2022 (this version, v3))
Abstract: Previous studies observed that finetuned models may be better base models than the vanilla pretrained model. Such a model, finetuned on some source dataset, may provide a better starting point for a new finetuning process on a desired target dataset. Here, we perform a systematic analysis of this intertraining scheme, over a wide range of English classification tasks. Surprisingly, our analysis suggests that the potential intertraining gain can be analyzed independently for the target dataset under consideration, and for a base model being considered as a starting point. This is in contrast to current perception that the alignment between the target dataset and the source dataset used to generate the base model is a major factor in determining intertraining success. We analyze different aspects that contribute to each. Furthermore, we leverage our analysis to propose a practical and efficient approach to determine if and how to select a base model in real-world settings. Last, we release an updating ranking of best models in the HuggingFace hub per architecture this https URL
Submission history
From: Leshem Choshen [view email][v1] Mon, 31 Oct 2022 19:24:02 GMT (5852kb,D)
[v2] Wed, 2 Nov 2022 08:49:45 GMT (5852kb,D)
[v3] Thu, 10 Nov 2022 18:06:28 GMT (5852kb,D)
Link back to: arXiv, form interface, contact.